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PREFACE 


This  document  is  Volume  II  of  the  Final  Technical  Report  (CDRL  A003)  for  the  Software 
Interoperability  and  Reusability  contract,  Number  F30602-80-C-0265.  The  contract  was 
performed  for  Rome  Air  Development  Center  (RADC)  to  provide  methodology  and 
technical  guidance  on  software  quality  metrics  to  Air  Force  software  acquisition 
managers. 

The  final  report  consists  of  two  volumes  as  follows: 

Volume  I  -  Final  Report  -  Software  Interoperability  and  Reusability 

Volume  II  -  Guidebook  for  Software  Quality  Measurement 

The  objective  of  this  contract  was  to  develop  techniques  to  measure  and  predict  software 
quality  with  a  perspective  on  interoperability  and  reusability.  The  techniques  developed 
were  to  be  assembled  into  a  "Handbook"  which  describes  the  step-by-step  procedures 
required  to  implement  the  quality  measurements.  Various  methods  of  assembling  a 
handbook  were  investigated  and  it  was  decided  that  the  best  approach  would  be  to  use  the 
"Software  Quality  Measurement  Manual"  (RADC-TR-80-109),  produced  by  a  prior  quality 
metric  research  contract,  as  the  baseline.  Volume  II  of  this  final  report  is  therefore  an 
update  of  RADC-TR-80-109,  incorporating  results  of  this  contract  and  the  results  of 
contract  F30602-80-C-0330,  "Quality  Metrics  for  Distributed  System".  In  addition,  many 
editorial  changes  and  corrections  were  made,  and  all  metric  worksheets,  tables,  and' 
definitions  are  included  as  appendices  so  that  all  material  required  to  implement  software 
quality  measurements  is  included  in  this  document. 

Volume  I  of  this  report  describes  the  results  of  the  research  effort  conducted  under  this 
contract. 
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SECTION  1 
INTRODUCTION 


1.1  PURPOSE 

There  has  been  an  increased  awareness  in  recent  years  of  the  critical  problems  that  have 
been  encountered  in  the  development  of  large  scale  software  systems.  These  problems 
include  not  only  the  cost  and  schedule  overruns  typical  of  development  efforts  and  the 
poor  performance  of  the  systems  once  they  are  delivered,  but  also  the  high  cost  of 
maintaining  the  systems,  the  lack  of  portability,  and  the  high  sensitivity  to  changes  in 
requirements. 

The  government  and  DoD  in  particular,  as  customers  of  many  large  scale  software  system 
developments,  have  sponsored  many  research  efforts  aimed  at  attacking  these  problems. 
For  example,  the  efforts  related  to  the  development  of  a  standard  DoD  programming 
language,  software  development  techniques,  and  development  tools  and  aids  all  provide 
partial  solutions  to  the  above  problems  by  encouraging  a  more  disciplined  approach  to  the 
development  of  software  and  therefore  a  more  controlled  development  process. 

A  related  research  thrust  which  has  been  recently  funded  by  DoD  is  the  area  of  software 
quality  metrics.  The  research  in  this  area  has  resulted  in  the  development  and  validation 
of  a  number  of  metrics  which  quantitatively  measure  various  attributes  of  software  which 
are  related  to  different  aspects  of  software  quality. 

The  potential  of  the  software  metric  concepts  can  be  realized  by  use  in  software 
procurement.  Their  use  enables  an  acquisition  manager  to  quantitatively  specify  the 
desired  level  of  quality  for  the  software  product  and  to  periodically  measure  the  achieved 
level  of  quality  throughout  the  software  development  process.  Their  effect  on  a  quality 
assurance  program  is  to  provide  a  more  disciplined,  engineering  approach  to  quality 
assurance  and  to  provide  a  mechanism  for  taking  a  life  cycle  viewpoint  of  software 
quality.  The  benefits  derived  from  their  application  are  realized  in  life  cycle  cost 
reduction  and  improved  soft  '  -  e  quality  resulting  from  added  visibility  for  management 
control. 

^The  purpose  of  this  guidebook  is  to  present  a  complete  set  of  procedures  and  guidelines 
for  introducing  and  utilizing  current  software  quality  metric  techniques  for  a  software 


procurement  associated  with  large  scale  software  system  developments.  These  proced¬ 
ures  and  guidelines  encompass: 

1.  ^  How  to  identify  and  specify  software  quality  requirements; 

2.  How  anc  when  to  apply  software  metrics;  and 

3.  How  to  interpret  the  information  obtained  from  the  application  of  the  metrics. 

1.2  SCOPE 

This  guidebook  incorporates  the  results  of  research  conducted  in  support  of  Rome  Air 
Development  Center  (RADC)  in  ihe  area  of  quality  metrics  for  distributed  systems  and 
software  interoperability  and  reusability.  It  is  an  update  of  the  "Software  Quality 
Measurement  Manual"  previously  produced  under  contract  number  F30602-78-C-0216  and 
published  as  RADC-TR-80-109,  Volume  II  (of  two).  Software  quality  metric  information 
for  the  quality  factors  of  survivability,  expandability,  interoperability  and  reusability  has 
been  added;  information  for  use  with  distributed  systems  has  been  added;  editorial 
changes  have  been  made;  the  metric  worksheets  have  been  refined,  reorganized,  and 
placed  in  an  appendix:  and  metric  tables  and  definitions  have  been  added  to  the  guidebook 
(appendices)  for  ease  of  use. 

While  some  aspects  of  the  technology  of  software  quality  metrics  require  further 
research,  those  portions  which  can  currently  provide  benefit  to  a  software  acquisition 
manager  are  emphasised  in  this  guidebook.  Guidelines  and  procedures  for  using  the 
software  metrics  are  described.  The  guidelines  and  procedures  are  presented  in  such  a 
way  as  to  facilitate  their  application  when  using  this  guidebook  for  a  software  develop¬ 
ment  project.  All  of  the  procedures  are  described  as  manual  processes,  however,  where 
automated  software  tools  could  be  used  to  compliment  or  enhance  the  process,  the  tools 
are  identified. 

Throughout  this  document  the  terms  guidebook,  handbook  and  manual  are  used  i  un¬ 
changeably. 


1.3  QUALITY  MEASUREMENT  IN  PERSPECTIVE 

The  evolution  during  the  past  decade  of  modern  programming  practices,  structured, 
disciplined  development  techniques  and  methodologies,  and  requirements  for  more  struc¬ 
tured,  effective  documentation  has  increased  the  feasibility  of  effective  measurement  of 
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software  quality.  However,  before  the  potential  of  measurement  techniques  could  be 
realized  a  framework  or  model  of  software  quality  had  to  be  constructed.  An  established 
model,  which  at  one  level  provides  a  user  or  management  oriented  view  of  quality,  is 
described  in  Section  2  of  this  guidebook  in  the  perspective  of  how  it  can  be  used  to 
establish  software  quality  requirements  for  a  specific  application. 

The  actual  measurement  of  software  quality,  described  in  Section  9.0,  is  accomplished  by 
applying  software  metrics  (or  measurements)  to  the  documental and  source  code 
produced  during  software  envelopment.  These  measurements  are  part  of  the  established 
model  of  software  quality,  and  through  that  model  they  can  be  related  to  various  user- 
oriented  aspects  of  software  quality. 

The  metrics  can  be  classified  according  to  three  categories:  anomaly-detecting, 
predictive,  and  acceptance. 

.  Anomaly-detecting  metrics  identify  deficiencies  in  documentation  or  source 
code.  These  deficiencies  usually  are  corrected  to  improve  the  quality  of  the 
software  product.  Standards  enforcement  is  a  form  of  anomaly-detecting 
metrics. 

.  Predictive  metrics  are  measurements  of  the  soundness  of  the  design  and 
implementation.  These  measurements  are  concerned  with  form,  structure, 
density,  and  complexity  type  attributes.  They  provide  an  indication  of  the 
quality  that  will  be  achieved  in  the  end  product-based  on  the  nature  of  the 
application  and  the  design  and  implementation  strategies. 

.  Acceptance  metrics  are  measurements  that  are  applied  to  the  end  product  to 
assess  the  final  compliance  with  requirements.  Tests  are  a  form  of 
acceptance-type  measurements. 

The  metrics  described  and  used  in  this  guidebook  are  either  anomaly-detecting  or 
predictive.  They  are  applied  during  the  software  development  phases  to  assist  in 
identification  of  quality  problems  early  in  the  life  cycle  so  that  corrective  actions  can  be 
taken  early  when  they  are  more  effective  and  economical  and  to  enable  a  prediction  of 
the  quality  level  expected  for  the  final  product. 
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The  measurement  concepts  complement  current  quality  assurance  practices;  they  are  not 
a  replacement  for  current  techniques  utilized  in  nornal  quality  assurance  programs.  For 
example,  a  major  objective  of  quality  assurance  is  to  assure  compliance  with  user/ 
customer  requirements.  The  software  quality  metric  concepts  described  in  this  guidebook 
provide  a  methodology  for  the  user/customer  to  specify  life-cycle-oriented  quality 
requirements,  usually  not  considered,  and  to  provide  a  mechanism  for  measuring  whether 
or  not  those  requirements  have  been  attained.  A  function  usually  performed  by  quality 
assurance  personnel  is  a  review/audit  of  software  products  produced  during  software 
development.  The  software  metrics  add  formality  and  quantification  to  these  document 
and  code  reviews.  The  metric  concepts  also  provide  a  vehicle  for  early  involvement  in  the 
development  process  since  there  are  metrics  which  apply  to  the  requirements  and  design 
documents  produced  early  in  the  development. 

Testing  is  usually  oriented  toward  evaluating  performance  (reliability,  usability,  perfor¬ 
mance,  efficiency,  etc.).  The  metrics  can  assist  in  evaluate  %  other  qualities  such  as 
maintainability,  portability,  flexibility,  etc. 

A  summary  of  how  the  software  metric  concepts  complement  quality  assurance  activities 
is  provided  in  Table  1.3-1.  This  is  based  on  the  quality  assurance  program  requirements 
identified  in  MIL-5-52779.  These  concepts  will  be  further  explained  and  illustrated  in  the 
subsequent  sections  of  this  guidebook. 

1.4  GUIDEBOOK  ORGANIZATION 

The  guidebook  has  been  organized  as  a  handbook  for  use  in  software  acquisition.  Section  1 
provides  introductory  information  and  how  the  guidebook  is  to  be  used. 

Section  2  defines  the  software  quality  model  and  describes  a  methodology  for  using  this 
model  to  establish  software  quality  requirements  or  goals  for  a  software  development 
project. 

Section  3  describes  procedures  for  measuring  the  quality  of  the  software.  These 
procedures  cover  what  to  measure,  when  to  measure,  and  how  to  measure. 

Section  4  describes  procedures  for  utilizing  the  information  provided  by  the  measurements 
to  make  assessments  of  the  quality  of  the  software  and  recommends  what  information  to 
present  to  various  personnel  involved  in  the  development. 
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Appendix  A  contains  the  metric  worksheets  used  for  collecting  data. 

Append'*  8  contains  the  metric  tables  used  for  calculating  metric  scores  during  the 
various  measurement  periods. 

Appendix  C  contains  a  detailed  description  of  the  metric  elements. 
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Table  1.3-1  How  Software  Metrics  Complement  Quality  Assurance 


MlL-S-52779 

QUALITY  ASSURANCE 

PROGRAM  REQUIREMENTS 

IMPACT  OF  SOFTWARE  QUALITY 

METRIC  CONCEPTS 

Assure  Compliance  with 

Requirements 

Adds  software  quality  requirements 

Identify  Software  Deficiencies 

Anomaly -detecting  metrics 

Provide  Configuration  Management 

No  impact 

Conduct  Test 

Assists  in  evaluation  of  other  qualities 

Provide  Library  Controls 

No  impact 

Review  Computer  Program  Design 

Predictive  metrics 

Assure  Software  Documentation 

Requirement  Compliance 

Metrics  assist  in  evaluation  of  documenta¬ 
tion  as  well  as  code 

Conduct  Reviews  and  Audits 

Procedures  for  applying  metrics  (in  form  of 
worksheets)  formalizes  inspection  process 

Provide  Tools/Techniques/Metho- 
dology  for  Quality  Assurance 

This  manual  describes  methodology  of  using 

metrics 

Provide  Subcontractor  Control 

Same  as  above  for  all  requirements 
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The  software  quality  metric  concepts  can  be  applied  at  several  levels.  In  an  acquisition 
manager/contractor  environment,  there  are  three  approaches  for  using  the  metric 
concepts.  They  are: 

1.  The  acquisition  manager's  staff  can  establish  software  quality  requirements  or 
goals  and  apply  metrics  to  the  delivered  software  products. 

2.  The  development  manager's  staff  can  apply  metrics  to  software  products 
during  development  and  report  them  to  the  acquisition  manager  during 
reviews. 

3.  An  independent  Quality  Assurance  or  Independent  Verification  and  Validation 
(IV&V)  contractor  can  apply  metrics  to  delivered  software  products  and  report 
them  to  the  acquisition  manager. 

Within  the  software  development  project  organization,  there  are  two  approaches  for  using 
the  metric  concepts.  They  are: 

1.  The  quality  assurance  personnel  can  apply  the  metrics  as  an  independent 
assessment  of  the  quality  of  the  software  being  produced  and  report  them  to 
the  software  development  manager. 

2.  The  software  development  personnel  can  apply  the  metrics  during  walk¬ 
throughs  and  reviews  and  report  them  to  the  software  development  manager. 

This  guidebook  is  oriented  toward  those  personnel  who  will  be  applying  the  quality  metrics 
concepts  (either  quality  assurance  or  development  personnel)  and  recommends  three 
approaches  to  both  establishing  the  quality  requirements  (Section  2)  and  making  a  quality 
level  assessment  (Section  4).  The  three  approaches  (an  index  is  provided  in  Table  1.5-1)  in 
each  area  are  presented  in  order  of  increasing  formality  of  the  relationship  between 
quality  requirements  and  metrics,  i.e.,  in  order  of  increasing  quantification.  The  order  of 
presentation  also  relates  to  an  increasing  requirement  for  experience  with  the  concepts  by 
the  personnel  applying  the  concepts.  Thus,  the  approaches  can  be  used  as  a  phased 
implementation  plan  of  incorporating  the  metric  concepts  into  the  quality  assurance 
functions. 
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Table  1.3-1  Index  of  Three  Approaches  to  Specifying  and  Assessing  Software  Quality 


“I 


APPROACH 

(LEVEL  OF 

FORMALITY) 

SPECIFYING 

SOFTWARE  QUALITY 

APPLYING 

MEASUREMENTS 

ASSESSING  THE 

QUALITY  OF 

THE  PRODUCT 

1 

Procedures  for  iden¬ 
tify  ing  important  quality 

factors 

(Paragraph  2.2) 

PROCEDURES 

FOR 

APPLYING 

Procedures  for 

the  inspector's 

assessment 

Procedures  for  iden¬ 
tifying  critical  software 

attributes 

(Paragraph  2.3) 

THE 

METRIC 

WORKSHEETS 

(SECTION  3) 

(Paragraph  4.2) 

2 

Procedures  for 

performing  sensi¬ 
tivity  analysis 
(Paragraph  4.3) 

3 

Procedures  for  establish¬ 
ing  quantifiable  goals 
(Paragraph  2.4) 

Procedures  for 

use  of  normaliza¬ 
tion  function 
(Paragraph  4.4) 

This  guidebook  is  recommended  to  the  personnel  applying  the  metric  concepts.  Additional 
information  and  definitions  can  be  found  in: 

"Factors  in  Software  Quality",  3  volumes,  RADC-TR-77-369,  Nov  1977.  (MCCA77) 
"Software  Quality  Metrics  Enhancements",  2  volumes,  RADC-TR-80-109,  April  1980 
"Softw :M*e  Interoperability  and  Reusability-Final  Report". 

"Software  Quality  Measurement  for  Distributed  Systems  -  Final  Report",  Volume  1 
of  this  report. 


These  references  are  recommended  to  the  personnel  applying  the  metrics  for  familiariza 
tion  with  the  underlying  concepts.  They  can  also  be  referred  to  periodically  for 
definitions  and  explanations. 
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SECTION  2 

IDENTIFYING  SOFTWARE  QUALITY  REQUIREMENTS 


2.1  INTRODUCTION 

The  primary  purpose  of  using  software  quality  metrics  in  a  software  acquisition  is  to 
improve  the  quality  of  the  software  product  by  specifying  software  quality  requirements 
and  by  measuring  and  predicting  achieved  software  quality.  The  concepts  can  improve 
quality  since  they  are  based  on  achieving  a  positive  influence  or  the  product. 

This  section  addresses  the  task  of  identifying  software  quality  requirements  and 
establishing  quantifiable  goals.  These  requirements  are  in  addition  to  the  functional, 
performance,  cost,  and  schedule  requirements  normally  specified  for  software  develop¬ 
ment.  The  fact  that  the  goals  established  are  related  to  the  quality  of  the  end  product 
should,  in  itself,  provide  some  positive  influence.  Past  research  has  shov/n  that  goal- 
directed  system  development  is  effective.  (WEIN72) 

The  vehicle  for  establishing  the  requirements  is  the  hierarchical  model  of  software  quality 
defined  in  (CAVA78).  This  model,  shown  in  Figure  2.1-1,  has  at  its  highest  level  a  set  of 
software  quality  factors  which  are  user /management-oriented  terms  and  represent  the 
characteristics  which  comprise  software  quality.  At  the  next  level,  for  each  quality 
factor,  there  is  a  set  of  criteria  which  are  the  attributes  that,  if  present  in  the  software, 
provide  the  characteristics  represented  by  the  quality  factors.  The  criteria,  then,  are 
software-related  terms.  Table  2.1-1  identifies  the  thirteen  quality  factors,  the  thirty 
quality  criteria,  and  their  relationships.  At  the  lowest  level  of  the  model  are  the  metrics 
which  are  quantitative  measures  of  the  software  attributes  defined  by  the  criteria.  In  a 
sense  there  is  a  still  lower  level  of  the  model  -  the  metric  elements.  Several  metric 
elements,  completed  at  several  points  in  the  software  life-cycle,  may  be  combined  in 
calculations  for  a  single  metric.  Appendix  B,  Metric  Tables,  identifies  the  metrics  and 
metric  elements. 

The  procedures  for  establishing  the  quality  requirements  for  a  particular  software  system 
utilize  this  model  and  are  described  as  a  three  level  approach;  the  levels  correspond  to  the 
hierarchical  levels  of  the  software  quality  model.  The  first  level  establishes  the  quality 
factors  that  are  important.  The  second  level  identifies  the  critical  software  attributes. 
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Tabic  2.1-1  Software  Quality  Factor*  and  Criteria 


SVSTf|>  A''CESS;RIUTY 


SYSTF.M  COMPATISIUTY 


VIRTUALITY 


VISIBIUTY 


The  third  level  identifies  the  metrics  that  will  be  applied  and  establishes  quantitative 
ratings  for  the  quality  factors. 

Once  the  quality  requirements  have  been  determined  by  following  the  procedures 
described  in  the  subsequent  paragraphs,  they  are  to  be  transmitted  to  the  development 
team.  In  a  formal  acquisition  manager /contractor  environment,  the  Request  for  Proposal 
(RFP)  is  the  medium  for  identifying  these  requirements.  The  results  of  following  the 
procedures  should  be  incorporated  in  the  RFP.  If  the  development  is  being  done 
internally,  the  quality  requirements  should  be  documented  in  the  same  form  as  the  other 
system  requirements.  A  briefing  emphasizing  the  intent  of  the  inclusion  of  the  quality 
requirements  can  also  be  conducted. 

2.2  IDENTIFYING  IMPORTANT  QUALITY  FACTORS 

2.2.1  Procedures 

The  basic  tool  utilized  in  identifying  the  important  quality  factors  is  the  Software  Quality 
Requirements  Form  shown  in  Table  2.2-1.  The  formal  definitions  of  each  of  the  thirteen 
factors  are  provided  on  that  form. 

A  briefing,  using  the  tables  and  figures  contained  in  this  paragraph,  should  be  conducted 
for  the  decision  makers  in  order  to  solicit  their  responses  to  the  survey.  The  decision 
makers  may  include  the  acquisition  manager,  the  user/customer,  the  development 
manager,  and  the  quality  assurance  manager.  To  complete  the  survey  the  following  five 
procedures  are  recommended. 

1  a.  Consider  Basic  Characteristics  of  the  Application 

The  software  quality  requirements  for  each  system  are  unique  and  are 
influenced  by  system  or  application-dependent  characteristics.  There  are 
basic  characteristics  which  affect  the  quality  requirements  and  each  software 
system  must  be  evaluated  for  its  basic  characteristics.  Table  2.2-2  provides  a 
list  of  some  of  these  basic  characteristics.  For  example,  if  the  system  is  being 
developed  in  an  environment  in  which  there  is  a  high  rate  of  technical 
breakthroughs  in  hardware  design,  portability  should  take  on  an  added  signifi¬ 
cance.  If  the  expected  life  cycle  of  the  system  is  long,  maintainability  and 
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Table  2.2-1  Software  Quality  Raquiramt.it*  Form 


The  13  quality  factor*  listed  balow  represent  aspects  of  software  quality  which  ara 
currently  thought  to  be  Important.  Indicate  whether  you  consider  each  factor  to  be 
Very  Important  (VO,  important  (0,  Somewhat  Important  (SI),  or  Not  Important  (NI)  a* 
design  goal*  in  the  tystem  you  are  currently  working  on  or  planning. 


pKEStSE 


FACTORS 

CORRECTNESS 

RELIABILITY 

EFFICIENCY 

INTEGRITY 

USADILIT'' 

SURVIVABILITY 

MAINTAINABlLfY 

VERIFIA3DL.TY 

FLEXIBILITY 

PORTABILITY 

REUSABILITY 

INTEROPERABILITY 

EXPANDABILITY 


DEFINITION 

Extent  to  which  the  software  aatisfies 
it*  specifications  and  fulfill*  the  user's 
mission  objectives. 

Probability  that  the  software  will  per¬ 
form  it*  logical  operations  in  the  speci¬ 
fied  environment  without  failure. 

Degree  of  utilization  of  resources  (pro¬ 
cessing  time,  storage,  communication 
time)  in  performing  functions. 

Extent  to  which  unauthorized  access  to 
the  software  or  data  can  be  controlled. 

Effort  for  training  and  software  opera- 


rung 

tion  -  familiarization,  input  preparation, 
execution,  output  interpretation. 

Probability  that  the  software  will  conti¬ 
nue  to  perform  or  support  critical  func¬ 
tions  when  a  portion  of  the  system  is 
inoperable. 

Average  effort  to  locate  ard  fix  a  soft¬ 
ware  failure. 

Effort  to  verify  the  specified  software 
operation  and  performance. 

Effort  to  extend  the  software  missions, 
functions,  or  data  to  satisfy  other 
requirements. 

Effort  to  convert  the  software  tor  use  in 
another  operating  environment  (hard¬ 
ware  configuration,  software  system 
environment). 

Effort  to  convert  a  software  component 
ter  use  in  another  application. 

Effort  to  couple  the  software  of  one 
system  to  the  software  of  another  sys¬ 
tem. 

Effort  to  Increase  software  capability  or 
performance  by  enhancing  current  func¬ 
tions  or  adding  new  functions/data. 


(Title* 


Name« 


Signature) 
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Table  2.2-2  Example  of  System  Characteristics  and  Related  Quality  Factors 


SYSTEM 

CHARACTERISTIC 

QUALITY  FACTOR 

If  human  lives  are  affected 

Reliability 

Correctness 

Verifiability 

Survivability 

Long  life  cycle 

Maintainability 

Expandability 

Experimental  system 
high  rate  of  change 

Flexibility 

High  technology  in  hardware  design 

Portability 

. 

Many  system  changes  over  life  cycle 

Reusability 

Expandability 

Real  time  application 

Efficiency 

Reliability 

Correctness 

On-board  computer  application 

Efficiency 

Reliability 

Correctness 

Survivability 

Processes  classified  information 

Integrity 

Interrelated  systems 

Interoperability 

expandability  become  cost-critica*  considerations.  If  the  application  is  an 
experimental  system  where  the  software  specifications  will  have  a  high  rate  of 
change,  flexibility  and  expandability  in  the  software  product  are  highly 
desirable.  If  the  functions  of  the  system  are  expected  to  be  required  for  a 
long  time,  while  the  system  itself  may  change  considerably,  reusability  and 
expandability  are  of  prime  importance  In  those  modules  which  implement  the 
major  functions  of  the  system.  With  the  advent  of  more  computer  networks 
and  communication  capabilities,  more  systems  are  being  required  to  interface 
with  other  systems  and  the  concept  of  interoperability  is  extremely  important. 
With  distributed  computing  systems,  more  attention  is  given  to  providing  some 
essential  computational  services  even  when  some  subsystems  are  inoperable, 
and  the  concept  of  survivability  is  extremely  important.  For  systems  with 
long  life-cycles  (e.g.,  15-20  years  for  a  major  weapon  system)  some  provisions 
must  be  made  for  incorporating  new  hardware  (add-on  memory  or  peripherals) 
or  new  software  (upgraded  operating  system),  and  the  concept  of  expandability 
becomes  crucial.  These  and  other  system  characteristics  should  be  considered 
when  identifying  the  important  quality  factors. 

If  system  level  quality  requirements  have  already  been  established,  refer  to 
Section  3.2  of  Volume  I  cf  this  report  for  aids  in  allocating  the  system  quality 
requirements  to  the  software  level  and  in  identifying  important  software 
quality  factors. 

lb.  Consider  Life  Cycle  Implications 

The  thirteen  quality  factors  identified  on  the  Software  Quality  Requirements 
Form  (Table  2.2-1)  can  be  grouped  according  to  three  life  cycle  activities 
associated  with  a  delivered  software  product.  These  three  activities  are 
product  operation,  product  revision,  and  product  transition.  The  relationship 
of  the  quality  factors  to  these  activities  is  shown  in  Table  2.2-3  under  the  post 
development  period.  This  table  also  illustrates  where  quality  ratings  can  be 
predicted  through  measurement  ( A )  and  where  the  impact  is  felt  if  poor 
quality  is  recognized  (X). 
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Legend:  *  When  Quality  h  Measured  and  Predicted,  X  *  When  Impact  of  Poor  Quality  »  Recognized 


The  size  of  this  impact  determines  the  cost  savings  that  can  be  expected  if  a  higher 
quality  system  is  achieved  through  the  application  of  the  metrics.  This  cost  savings  is 
somewhat  offset  by  the  cost  to  apply  the  metrics  and  the  cost  to  develop  the  higher 
quality  software  product  as  illustrated  in  Figure  2.2-1.  The  cost  to  apply  the  metric  is 
difficult  to  estimate  for  the  first  project  in  which  they  are  applied.  This  is  due  to  the 
training  time  for  personnel  applying  metrics.  Experience  shows  that  a  learning  curve 
applies  —  that  subsequent  applications  of  metrics  have  a  lower  cost  and  greater  cost 
saving  opportunities. 


COST  TO  DEVELOP 

LIFE 

CYCLE 
SAVINGS  AS 

HIGH  QUALITY  SOFTWARE 

A  RESULT  OF 

PLUS 

HIGHER 

COST  TO 

QUALITY 

MEASURE  QUALITY 

PRODUCT 

I 


Figure  2.2-1  Benefit  Tradeoff:  Quality  Costs  vs  Cost  Savings 

This  cost  to  implement  versus  life-cycle  cost  reduction  relationship  exists  for  each 
quality  factor.  The  benefit,  cost-to-provide  versus  cost-saved  ratio,  for  each  factor  is 
rated  as  high,  medium,  or  low  in  the  right  hand  column  of  Table  2.2-3.  This  relationship 
and  the  life-cycle  implications  of  the  quality  factors  should  be  considered  when  selecting 
the  important  factors  for  a  specific  system. 

lc.  Performance  Tradeoffs  Among  the  Quality  Factors 

As  a  result  of  steps  la  and  lb,  a  tentative  list  of  quality  factors  should  be 
produced.  The  next  step  is  to  consider  the  interrelationships  among  the 
factors  selected.  Tables  2.2-4  and  2.2-3  can  be  used  as  a  guide  for 
determining  the  relationships  between  the  quality  factors.  Some  factors  are 
synergistic  while  other  conflict.  The  impact  of  conflicting  factors  is  that  a 
lower  quality  level  can  be  expected  for  each  factor  if  both  are  required  than 
can  be  expected  if  only  one  or  the  other  is  required.  The  synergistic  (positive 
tradeoffs)  and  conflicts  (negative  tradeoffs)  may  reflect  a  more  complex 
interrelationship  of  factors.  For  example,  there  may  be  a  group  of  three 
factors  which  can  all  be  enhanced  together  by  a  design  decision.  An  effort 
should  be  made  to  identify  such  multiple  tradeoffs  for  the  particular  software 
product. 
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Table  2.2-4  Relationship*  Between  Software  Quality  Factors 

LEGEND: 

IF  A  HIGH  DEGREE  OF  QUALITY 
IS  PRESENT  FOR  ONE  FACTOR, 

THE  DEGREE  OF  QUALITY  EXPECTED 
FOR  THE  OTHER  FACTOR  IS: 


^  »  HIGH 
4N  a  LOW 


Table  2.2-5  Typical  Factor  Tradeoffs 


muamjty 

ifwobncy 

THE  ADDITIONAL  CODE  REQUIRED  TO  PROVIDE  ACCURACY  AMO 

TO  PERFORM  ANOMALY  MANAGEMENT  USUALLY  INCREASES  RUN 
TIME  AND  REQUIRES  ADDITIONAL  STORAGE 

VS 

FUXIBNJTY 

rbusabujiy 

CXTAMOAKUTY 

INTERORIRABlUTY 

THE  GENERALITY  REQUIRED  FOR  FLEXIBLE.  REUSABLE.  AND 
EXPANDABLE  SOFTWARE  USUALLY  INCREASES  THE  DIFFICULTY  OF 
PROVIDING  ACCURACY  AND  PERFORMING  ANOMALY 

MANAGEMENT  FOR  SPECIFIC  CASES. 

INTEGRITY 

THE  ADDITIONAL  CODE  AND  PROCESSING  REQUIRED  TO  CONTROL 
ACCESS  to  CODE  OR  DATA  USUALLY  LENGTHENS  RUN  TIME  AND 
REQUIRES  ADDITIONAL  STORAGE. 

USABILITY 

THE  ADDITIONAL  COOE  AND  PROCESSING  REQUIRED  to  EASE  AN 
OPERATOR'S  TASK  OR  TO  PROVIDE  MORE  USABLE  OUTPUT 

USUALLY  INCREASE  RUNTIME  AND  REQUIRE  ADDITIONAL 

STORAGE. 

SURVIVABILITY 

THE  AODITIONAl  COOE  AND  PROCESSING  REQUIREO  FOR 

MODULAR.  RECONFlGURABLE.  ANOMALY  TOLERANT  SOFTWARE 
RESULTS  IN  LESS  EFFICIENT  OPERATION 

EFFICIENCY 

VI 

MAINTAINABILITY 

VERIFIABILITY 

USING  MODULAR.  VISIBLE.  SELF-DESCRIPTIVE  COOE  TO  INCREASE 
MAINTAINABILITY  AND  VERIFIABILITY  USUALLY  INCREASES 
OVERHEAD  AND  RESULTS  IN  LESS  EFFICIENT  OPERATION  COOE 
WHICH  IS  OPTIMIZED  FOR  EFFICIENCY  POSES  PROBLEMS  TO  THE 
TESTER  BMAINTAINER. 

FLEXIBILITY 

REUSABILITY 

THE  GENERALITY  REQUIRED  FOR  FLEXIBLE  AND  REUSABLE 

SOFTWARE  INCREASES  OVERHEAD  AND  DECREASES  EFFICIENCY 

PORTABILITY 

THE  USE  OF  COOE  OPTIMIZED  FOR  EFFICIENCY  USUALLY 

OECREASES  PORTABILITY. 

INTIROPERABIUTY 

THE  OVERHEAO  FOR  CONVERSION  FROM  STANDARD  DATA 
REPRESENTATIONS  AND  FOR  THE  USE  OF  STANDARD  INTERFACE 
ROUTINES  OECREASES  OPCRAT'NG  EFFICIENCY 

EXRANOABlUTY 

THE  USE  OF  MODULAR.  GENERAL  SOFTWARE  USUALLY  DECREASES 
OPERATING  EFFICIENCY 

SURVIVABILITY 

THE  OISTRIBUTEONESS  REQUIRED  FOR  SURVIVABLE  SOFTWARE 
INCREASES  THE  RISK  Of  UNAUTHORIZED  ACCESS. 

Fl.EXHIl.ITY 

REUSABILITY 

THE  GENERALITY  REQUIRED  FOR  FLEXIBLE  ANO  REUSABLE 

SOFTWARE  INCREASES  THE  RISK  OF  UNAUTHORIZED  ACCESS 

INTEGRH  v 

vs 

INTERORERABIU  rY 

COURLEP  SYSTEM  HAVE  MORE  AVENUES  OF  ACCESS.  DIFFERENT 
USERS,  ANO  COMMON  DATA  REPRESENTATIONS;  THEY  OFTEN 

SHARE  DATA  AND  COOE.  THESE  INCREASE  THE  POTENTIAL  FOR 
ACCIDENTAL  OR  OEl'BERATE  ACCESS  OF  SENSITIVE  DATA. 

EXPANDABILITY 

THE  GENERALITY  REQUIRED  FOR  EXPANDABLE  SOFTWARE 

INCREASES  THE  RISK  OF  UNAUTHORIZED  ACCESS. 

survivability 

VI 

FLEXIBILITY 

PORTABILITY 

REUSABILITY 

THE  RECONFIGURABILITY  REQUIRED  FOR  SURVIVALS  SOFTWARE 
REDUCES  ITS  FLEXIBILITY,  PORTABILITY,  AND  REUSA8IUTY. 

INTfROPtRABIUTY 

VS 

MAINTAINABILITY 

VER1FIAJIUTY 

FLEXIBILITY 

THE  ADDITIONAL  COMPLEXITY  INTRODUCED  BY 

COMMUNICATION.  FUNCTIONAL  INTERPACING.  ANO  DATA 
COMMONALITY  BETWEEN  SYSTEMS  INCREASES  THE  COMPLEXITY 

OF  CHANGING,  VERIFYING.  AND  MAINTAINING  THE  SOFTWARE 

ld.  Identify  Most  Important  Quality  Factors 

Based  on  ia  through  lc,  a  list  of  quality  factors  considered  to  be  important  for 
the  particular  system  can  be  compiled.  The  list  should  be  organized  in  order 
of  importance.  A  single  decision  maker  can  be  assigned  to  choose  the  factors 
or  the  choice  can  be  made  by  averaging  several  survey  responses.  The 
definitions  of  the  factors  chosen  should  be  included  with  this  list. 

le.  Provide  Explanation  for  Choice 

The  rationale  for  the  decisions  made  during  steps  la  through  lc  should  be 
documented.  If  a  factor  is  not  considered  important  for  the  system,  a 
rationale  may  also  be  provided.  For  example,  maintainability  may  not  be 
emphasized  because  verifiability  (given  top  priority)  will  ensure  a  thoroughly 
tested  (and  therefore  highly  maintainable)  product. 

2.2.2  An  Example  of  Factors  Specification 

To  illustrate  the  application  of  the  above  steps,  consider  a  spare  parts  inventory  control 
system.  The  inventory  control  system  maintains  inventory  status  and  facilitates  requisi¬ 
tioning,  reordering,  and  issuing  of  spare  parts  to  Air  Force  units  in  support  of  various 
missions.  The  planned  life  of  the  system  is  ten  years. 

Each  step  described  previously  will  be  performed  with  respect  to  the  spare  parts  inventory 
control  system. 

la.  Consider  Basic  Characteristics  of  the  Application 

Utilizing  Table  2.2-2  and  considering  the  unique  characteristics  of  the  spare 
parts  inventory  control  system  resulted  in  the  following: 

Characteristic  Related  Quality  Factor 

Critical  Support  for  Reliability 

a  Flying  Unit  Correctness 

Verifiability 

Survivability 
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Characteristic 
Long  Life  Cycle 
With  Stable  Hardware 
And  Software  Requirements 

Utilized  By  Air  Force  Main¬ 
tenance  Personnel 


Related  Quality  Factor 
Maintainability 


Usability 


Interfaces  with  other  Air  Interoperability 

Force  Inventory  Systems  (e.g. 

Supplies) 


lb.  Consider  Life  Cycle  Implications 

For  the  five  quality  factors  identified  in  la,  determine  the  life  cycle  cost 
benefits  according  to  Table  2.2-3. 


QUALITY 

FACTORS 

Reliability 

Correctness 

Verifiability 

Survivability 

Maintainability 

Usability 

Interoperability 


COST  BENEFIT  RATIO 
High 
High 
High 
Low 
High 
Medium 
Medium 


lc.  Perform  Trade  Offs  Among  Quality  Factors 

Using  Table  2.2-4,  there  are  no  conflicts  which  need  to  be  considered. 


Id.  Identify  Most  Important  Quality  Factors 

Using  Table  2.2-1  and  the  guidance  provided  by  steps  la  through  lc,  the 
following  factors  are  identified  in  order  of  importance;  provide  the  definitions. 

CORRECTNF.SS  -Extent  to  which  the  software  satisfies  its  specifica¬ 
tions  and  fulfills  the  user's  mission  objectives. 
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RELIABILITY 


.Probability  that  the  software  wiii  perform  its  logical 
operations  in  the  specified  environment  without  fail¬ 
ure. 

USABILITY  -Effort  for  training  and  software  operation  -tamiliari 

zation,  input  preparation,  execution,  output  interpre¬ 
tation. 

VERIFIABILITY  -Effort  to  verify  the  specified  software  operation  and 

performance. 

SURVIVABILITY  -Probability  that  the  software  will  continue  to  perform 

or  support  critical  functions  when  a  portion  of  the 
system  is  inoperable. 

MAINTAINABILITY  -Average  effort  to  locate  and  fix  a  software  failure. 

INTEROPERABILITY  -Effort  to  couple  the  software  of  one  system  to  the 

software  of  another  system. 

le.  Provide  Explanation  for  Choice 

Document  the  rationales  for  the  decisions  made  in  the  above  step. 

CORRECTNESS  -System  performs  critical  spare  parts  provision 

function. 

RELIABILITY  -System  performs  critical  spar^  parts  provision 

functions  in  field  environment. 

VERIFIABILITY  -System  performs  critical  spare  parts  provision 

functions. 

SURVIVABILITY  -System  performs  critical  spare  parts  provision 

function  in  field  environment  and  will  interface  with 
other  systems. 
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USABILITY  -System  will  be  used  by  military  personnel  with  mini¬ 

mum  computer  training. 

MAINTAINABILITY  -System  life  cycle  is  projected  to  be  10  years  and  will 

operate  in  the  field  and  be  maintained  by  military 
personnel. 

INTEROPERABILITY  -System  will  interface  with  other  inventory  systems. 

2.3  IDENTIFYING  CRITICAL  SOFTWARE  ATTRIBUTES 

2.3.1  Procedures 

The  second  level  of  identifying  the  quality  requirements  involves  proceeding  from  the 
user-oriented  quality  factors  to  the  software-oriented  criteria.  Sets  of  criteria,  which  are 
attributes  of  the  software,  are  related  to  the  various  factors  by  definition.  Their 
identification  is  automatic  and  represents  a  more  detailed  specification  of  the  quality 
requirements.  Identification  of  a  quality  factor  does  not  automatically  mean  that  all 
criteria  within  tnat  factor  are  equally  important.  Tradeoffs  and  synergisms  may  exist 
between  criteria  within  the  same  factor.  A  subset  of  the  criteria  within  a  factor  may  be 
identified. 


2a.  Identify  Critical  Software  Attributes  Required 

Table  2.3-1  is  used  to  identify  the  software  attributes  (criteria)  associated 
with  the  chosen  software  quality  factors. 


j 

i 

i 
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Table  2.3-1  Software  Criteria  and  Related  Quality  Factors 


QUALITY 

FACTOR 

SOFTWARE  CRITERIA 

CORRECTNESS 

COMPLETENESS 

CONSISTENCY 

SIMPLICITY 

SPECIFICITY 

TRACEABILITY 

EFFICIENCY 

EFFECTIVENESS 

FLEXIBILITY 

GENERALITY 

MODULARITY 

SELF-DESCRIPTIVENESS 

SIMPLICITY 

INTEGRITY 

SYSTEM  ACCESSIBILITY 

VIRTUALITY 

INTEROPERABILITY 

AUGMENT  ABILITY 

commonality 

COMMUNICATIVENESS 

FUNCTIONAL  OVERLAP 

INDEPENDENCE 

MODULARITY 

SYSTEM  COMPATIBILITY 

MAINTAINABILITY 

CONCISENESS 

CONSISTENCY 

MODULARITY 

SELF-DESCRIPTIVENESS 

SIMPLICITY 

VISIBILITY 
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Table  2.3-1  (continued) 


r 


QUALITY 

FACTOR 


SOFTWARE  CRITERIA 


EXPANDABILITY 


AUGMENT  ABILITY 

GENERALITY 

MODULARITY 

SIMPLICITY 

SPECIFICITY 

VIRTUALITY 


PORTABILITY 


RELIABILITY 


REUSABILITY 


VERIFIABILITY 


INDEPENDENCE 

MODULARITY 

SELF-DESCRIPTIVENESS 


ACCURACY 

ANOMALY  MANAGEMENT 

CONSISTENCY 

SIMPLICITY 


APPLICATION  INDEPENDENCE 

DOCUMENT  ACCESSIBILITY 

FUNCTIONAL  SCOPE 

GENERALITY 

INDEPENDENCE 

MODULARITY 

SELF  DESCRIPTIVENESS 

SIMPLICITY 

SYSTEM  CLARITY 


MODULARITY 

SELF  -DESCRIPTIVENESS 

SIMPLICITY 

SPECIFICITY 

VISIBILITY 


Table  2.3-1  (continued) 


FACTOR^ _ _ 

SOFTWARE  CRITERIA 

USABILITY 

COMMUNICATIVENESS 

OPERABILITY 

TRAINING 

VIRTUALITY 

VISIBILITY 

SURVIVABILITY 

ANOMALY  MANAGEMENT 

AUTONOMY 

DISTRIBUTEDNESS 

MODULARITY 

RECONFIGURABILITY 

2b.  Provide  Definitions 

Table  2.3-2  should  be  used  to  provide  the  definitions  of  criteria  as  part  of  the 
specification. 


Table  2.3-2  Definitions  of  Software  Criteria 


SOFTWARE  CRITERION 

DEFINITION 

ACCURACY 

Those  attributes  of  the  software  which  provide 
the  required  precision  in  calculations  and  outputs. 

ANOMALY  MANAGEMENT 

Those  attributes  of  the  software  which  provide 
for  continuity  of  operations  under,  and  recovery 
from  nonnominai  conditions. 

APPLICATION  INDEPENDENCE 

Attributes  of  the  software  which  determine  its 
dependency  on  the  software  application  (database 
system,  data  structure,  system  libraries  routines, 
microcode,  computer  architecture  and  algorithms) 

AUGMENTABILITY 

Those  attributes  of  the  software  which  provide 
for  expansion  of  capability  for  functions  and  data. 

AUTONOMY 

Those  attributes  of  the  software  which  determine 
its  nondependency  on  interfaces  and  functions. 

COMMONALITY 

Those  attributes  of  the  software  which  provide 
for  the  use  of  interface  standard  for  protocols, 
routines,  and  data  representations. 

COMMUNICATIVENESS 

Those  attributes  of  the  software  which  provide 
useful  inputs  and  outputs  which  can  be  assimila¬ 
ted. 

COMPLETENESS 

Those  attributes  of  the  software  which  provide 
full  implementation  of  the  functions  required. 

CONCISENESS 

Those  attributes  of  the  software  which  provide 
for  implementation  of  a  function  with  a  minimum 
amount  of  code. 

CONSISTENCY 

Those  attributes  of  the  software  which  provide 
for  uniform  design  and  implementation  techniques 
and  notation. 

DISTRIBUTEDNESS 

Those  attributes  of  the  software  which  determine 
the  degree  to  which  software  functions  are  geo¬ 
graphically  or  logically  separated  within  the  sys¬ 
tem. 

DOCUMENT  ACCESSIBILITY 

Attributes  of  the  software  which  provide  easy 
access  to  and  selective  use  of  system  components. 
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Table  2.3-2  (continued) 


SOFTWARE  CRITERION 

DEFINITION 

EFFECTIVENESS 

Those  attributes  of  the  software  which  provide 
for  minimum  utilization  of  resources  (processing 
time,  storage,  operator  time)  in  performing  func¬ 
tions. 

FUNCTIONAL  OVERLAP 

A  comparison  between  two  systems  to  determine 
the  number  of  functions  common  to  both  systems. 

FUNCTIONAL  SCOPE 

Those  attributes  of  the  soitware  which  provide 
the  scope  of  functions  required  to  be  performed 
i.e.  specificity,  commonality  and  completeness. 

GENERALITY 

Those  attributes  of  the  software  which  provide 
breadth  to  the  functions  performed  with  respect 
to  the  application. 

INDEPENDENCE 

Those  attributes  of  the  software  which  determine 
its  non-dependency  on  the  software  environment 
(computing  system,  operating  system,  utilities, 
input/output  routines,  libraries). 

MODULARITY 

Those  attributes  of  the  software  which  provide  a 
structure  of  highly  cohesive  modules  with  opti¬ 
mum  coupling. 

OPERABILITY 

Those  attributes  of  the  software  which  determine 
operations  and  procedures  concerned  with  the 
operation  of  the  software. 

RECONFIGURABILITY 

ihose  attributes  of  the  software  which  provide 
for  continuity  of  system  operation  when  one  or 
more  processors,  storage  units,  or  communication 
links  fail. 

SELF  -DESCRIPTIVENESS 

Those  attributes  of  the  software  which  provide 
explanation  of  the  implementation  of  a  function. 

SIMPLICITY 

Those  attributes  of  the  software  which  provide 
for  the  definition  and  implementation  of  functions 
in  the  most  non-complex  and  understandable  man¬ 
ner. 

SPECIFICITY 

Those  attributes  of  the  software  which  provide 
for  singularity  in  the  definition  and  implementa¬ 
tion  of  functions. 

SYSTEM  ACCESSIBILITY 

Those  attributes  of  the  software  which  provide 
for  control  and  audit  of  access  of  software  and 
data. 

Table  2.3*2  (continued) 


SOFTWARE  CRITERION 

DEFINITION  _  _ 

SYSTEM  CLARITY 

Those  attributes  of  the  software  which  provide 
clear  description  of  program  structure  in  the  most 
non-complex,  easily  understandable  and  modi¬ 
fiable  manner. 

SYSTEM  COMPATIBILITY 

A  measure  of  the  hardware,  software  and  com¬ 
munication  compatibility  of  two  systems. 

TRACEABILITY 

Those  attributes  of  the  software  which  provide  a 
thread  of  origin  from  the  implementation  to  the 
requirements  with  respect  to  the  specific  devel¬ 
opment  envelope  and  operational  environment. 

TRAINING 

Those  attributes  of  the  software  which  provide 
transition  from  current  operation  or  provide 
initial  familiarization. 

VIRTUALITY 

Those  attributes  of  the  software  which  present  a 
system  that  does  not  require  user  knowledge  of 
the  physical,  logical,  or  topological  characteris¬ 
tics  (e.g.,  number  of  processors/di3ks,  storage 
locations). 

VISIBILITY 

Those  attributes  of  the  software  which  provide 
status  monitoring  of  the  development  and  opera¬ 
tion  (e.g.,  instrumentation). 
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2.3.2 


Example  of  Identifying  Software  Criteria 


Continuing  with  the  example  of  paragraph  2.2.2,  the  software  criteria  for  the  identified 
quality  factors  would  be  chosen. 

2a.  Identify  Critical  Software  Attributes 

Using  the  relationships  provided  in  Table  2.3-1,  the  software  criteria  shown  in 
Table  2.3-3  would  be  identified.  Evaluation  of  the  definitions  of  the  criteria  in 
the  context  of  the  software  product  and  its  quality  goals,  may  allow  a  number 
of  the  resulting  criteria  to  be  eliminated. 


Table  2.3-3  Software  Criteria  to  Factor  Relationships 


RELATED  FACTOR 

SOFTWARE 

CRITERIA 

CO 

RL 

SV 

MA 

VE 

US 

IP 

TRACEABILITY 

X 

CONSISTENCY 

X 

X 

X 

COMPLETENESS 

X 

ANOMALY  MANAGE- 

X 

X 

MENT 

ACCURACY 

X 

SIMPLICITY 

X 

X 

X 

CONCISENESS 

X 

MODULARITY 

X 

X 

X 

X 

SELF-DESCRIPTIVENESS 

X 

X 

OPERABILITY 

X 

TRAINING 

X 

COMMUNICATIVENESS 

X 

COMMONALITY 

X 

FUNCTIONAL  OVERLAP 

X 

INDEPENDENCE 

X 

SYSTEM  COMPATIBILITY 

X 

VISIBILITY 

X 

X 

X 

AUGMENTABILITY 

X 

MODULARITY 

X 

AUTONOMY 

X 

DISTRIBUTEDNESS 

X 

RECONFIGURABILITY 

X 

SPECIFICITY 

X 

CO  =  Correctness,  RL  =  Reliability,  SV  =  Survivability 
MA  =  Maintainability,  VE  =  Verifiability, 

US  =  Usability,  IP  =  Interoperability 
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2b.  Provide  Definitions 

The  definitions  for  each  of  these  software  criteria,  as  shown  in  Table  2.3-2 

would  also  be  provided  as  part  of  the  specification. 

2.4  ESTABLISHING  QUANTIFIABLE  GOALS 

2.4.1  Procedures 

The  third  and  last  level,  which  is  the  most  detailed  and  quantified,  requires  precise 
statements  of  the  level  of  quality  that  will  be  acceptab ie  for  the  software  product. 

Currently,  the  underlying  mathematical  relationships  which  allow  measurement  at  this 
level  of  precision  do  not  exist  for  all  of  the  quality  factors.  The  mechanism  for  making 
the  precise  statement  for  any  quality  factor  is  a  rating  or  figure-of-merit  of  the  factor. 
The  underlying  basis  for  the  ratings  of  all  factors  except  reliability  and  survivability  is  the 
effort  or  cost  required  to  perform  a  function  such  as  to  correct  or  modify  the  design  or 
program.  For  example,  rating  for  maintainability  might  be  that  the  average  time  to  fix  a 
problem  should  be  five  man-days  or  that  90%  of  the  problem  fixes  should  take  less  than 
six  man-days.  This  rating  would  be  specified  as  a  quality  requirement.  To  comply  with 
this  specification,  the  software  would  have  to  exhibit  characteristics  which,  when  present, 
give  an  indication  that  the  software  will  perform  to  this  rating.  These  characteristics  are 
measured  by  metrics  which  are  inserted  into  a  mathematical  relationship  to  obtain  the 
predicted  rating.  Note  that  the  reliability  ratings  are  provided  in  terms  familiar  to 
traditional  hardware  reliability.  Oust  as  in  hardware  reliability  there  are  significant 
differences  between  ratings  of  .9  and  .99. 

In  order  to  choose  ratings  such  as  the  two  mentioned  above,  data  must  be  available  which 
allows  the  decision  maker  to  know  what  is  a  "good  rating"  or  perhaps  what  is  the  industry 
average.  Currently  there  is  generally  a  lack  of  good  historical  data  to  establish  these 
expected  levels  of  operations  and  maintenance  performance  for  software.  There  are 
significant  efforts  underway  to  compile  historical  data  and  derive  the  associated 
performance  statistics  (DUVA76).  Individual  software  development  organizations  and 
System  Program  Offices  should  attempt  to  compile  historical  data  for  their  particular 
environment.  Any  environment-unique  data  available  should  be  used  as  a  check  against 
the  data  provided  as  guidelines  in  this  manual.  The  data  utilized  in  this  section  is  based 
on  experiences  applying  the  metrics  to  several  large  command  and  control  software 
systems  and  other  experiences  reported  in  the  literature. 
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3a.  Specify  Rating  for  Each  Quality  Factor 

After  identification  of  the  critical  quality  factors,  specific  performance  levels 
or  ratings  required  for  each  factor  should  be  specified.  Tables  2.4-1  and  2.4-2 
should  be  used  as  a  guideline  for  identifying  the  ratings  for  the  particular 
factors.  Note  that  mathematical  relationships  have  not  been  established  for 
some  of  the  factors.  In  those  cases,  it  is  advisable  not  to  levy  requirements 
for  meeting  a  specific  quality  rating  but  instead  specify  the  relative  impor¬ 
tance  (priority)  of  the  quality  factor  as  a  development  goal. 


3b.  Identify  Specific  Metrics  to  be  Applied 

The  next  step  or  an  alternative  to  3a  is  to  identify  the  specific  metrics  which 
will  be  applied  to  the  various  software  products  produced  during  the  develop¬ 
ment.  The  Metric  Worksheets  described  in  Appendix  A  can  be  used  for  this 
purpose  or  Table  2.4-3  can  be  used  to  identify  the  metrics  and  reference  can 
be  made  to  Appendix  C  where  definitions  of  the  metrics  are  provided. 
Detailed  examination  may  allow  a  subset  of  the  metrics  within  a  criteria  to  be 
isolated. 


3c.  Specification  of  Metric  Threshold  Values 

In  lieu  of  specifying  quality  ratings  or  in  addition  to  the  ratings,  specific 
minimum  values  for  particular  metrics  may  be  specified.  This  technique  is 
equivalent  to  establishing  a  standard  which  is  to  be  adhered  to.  Measurements 
less  than  the  value  established  are  to  be  reported.  Typical  values  can  be 
derived  by  applying  the  metrics  to  software  products  developed  in  a  particular 
environment  or  by  looking  at  the  scores  reported  in  (MCCA77J,  (MCCA80)  or 
Volume  1  of  this  report.  When  establishing  these  threshold  values  based  on 
past  project  data,  projects  which  have  been  considered  successful,  i.e.,  have 
demonstrated  good  characteristics  during  their  life  cycle  should  be  chosen. 
For  example,  a  system  which  has  been  relatively  cost-effective  to  maintain 
over  its  operational  history  should  be  chosen  and  the  metrics  related  to 
maintainability  applied  to  establish  threshold  values.  Incentives  may  also  be 
offered  if  a  particular  metric  exceeds  a  maximum  threshold  value. 
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Table  2*4-1  Quality  Factor  Ratings 


QUALITY  FACTOR 

RATING  EXPLANATION 

RATING  GUIDELINES 

RELIABILITY* 

Rating  is  in  terms  of  the  number 

RATING 

.9 

.98** 

.99 

.999 

of  errors  that  occur  after  the 
start  of  formal  testing. 

Rating  =  1- Number  of  Errors 
Number  of  Lines  of 
source  code  exclud¬ 
ing  comments 

ERRORS 

10 

2 

1 

.1 

■ 

MAINTAINA¬ 

BILITY* 

Rating  is  in  terms  of  the  average 
amount  of  effort  required  to  lo¬ 
cate  and  fix  an  error  in  an  opera¬ 
tional  program. 

Rating  =  l-.l  (Average  number 
of  man  days 
per  fix) 

RATING 

.5 

mm 

.9 

AVERAGE 

EFFORT 

(MAN 

DAYS) 

1 

5 

3 

1 

PORTABILITY* 

Rating  is  in  terms  of  the  effort 
required  to  convert  a  program  to 
run  in  another  environment  with 
respect  to  the  effort  required  to 
originally  implement  the  program. 

Rating  =  1 -Effort  to  Transoort 
Effort  to  Implement 

RATING 

.25 

.5** 

.75 

.9 

%  OF 

ORIGINAL 

EFFORT 

75 

50 

25 

10 

FLEXIBILITY* 

Rating  is  in  terms  of  the  average 
effort  required  to  extend  a  pro¬ 
gram  to  include  other  require¬ 
ments. 

Rating  =  l-.05(Average  number 

of  man  days  to  change) 

RATING 

.3 

.5** 

B 

.9 

AVERAGE 

EFFORT 

(MAN 

DAYS) 

14 

10 

6 

2 

REUSABILITY* 

Rating  is  in  terms  of  the  effort 
required  to  convert  a  program  to 
a  different  application  with  re¬ 
spect  to  the  effort  required  to 
build  a  new  program. 

Rating  =  1 -Effort  to  Convert 
Effort  to  Build 

RATING 

.2 

■S3 

.75 

.9 

%  OF 
EFFORT 
TO  BUILD 

80 

60 

25 

10 

Table  2.4-1  (Continued) 


QUALITY  FACTOR 

RATING  EXPLANATION 

RATING  GUIDELINES 

INTEROPERA¬ 

BILITY* 

Rating  is  in  terms  of  the  effort 
required  to  couple  the  system  to 
another  system. 

Ratine  «  1- Effort  to  Modify 

Effort  to  Build 

RATING 

.2 

n 

.75 

H 

%  OF 
EFFORT 

TO  BUILD 

SO 

50 

25 

10 

EXPANDABILITY* 

Rating  is  in  terms  of  the  effort 
to  increase  software  capability, 
performance  and  original  devel¬ 
opment  effort. 

RATING 

.4 

.5 

n 

n 

%  OF 
EFFORT 

TC  DEVELOI 

SO 

45 

30 

10 

NOTES 

*  Dm  collected  to  date  provides  some  basis  upon  which  to  allow  quantitative 
ratings  (or  these  quality  factors.  These  ratings  should  be  modified  based  on 
data  collected  within  a  specific  development  environment.  Data  has  not  been 
collected  to  support  ratings  of  the  other  quality  factors. 

**  Indicates  rating  which  might  be  considered  current  industry  average. 
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Table  2.4-2  Quality  Factor  Rating  Explanation 


QUALITY 

FACTOR 

RATING  EXPLANATION 

(Guidelines  Not  Established) 

CORRECTNESS 

The  function  which  the  software  is  to  perform  is  incorrect.  The 
rating  is  in  terms  of  effort  required  to  implement  the  correct 

function. 

EFFICIENCY 

The  software  does  not  meet  performance  (speed,  storage)  require¬ 
ments.  The  rating  is  in  terms  of  effort  required  to  modify 
software  to  meet  performance  requirements. 

INTEGRITY 

The  software  does  not  provide  required  security.  The  rating  is  in 
terms  of  effort  required  to  implement  proper  levels  of  security. 

USABILITY 

There  is  a  problem  related  to  operation  of  the  software,  the  user 
interface,  or  the  input/output.  The  rating  is  in  terms  of  effort 
required  to  improve  human  factors  to  acceptable  level. 

VERIFIABILITY 

The  rating  i§  in  terms  of  effort  required  to  test  changes  or  fixes. 

SURVIVABILITY 

The  rating  is  in  terms  of  the  number  of  survivability  related  errors 
that  occur  after  the  start  of  formal  testing. 
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Table  2.4-3  Quality  Metrics  Related  to  Factors 


QUALITY 

FACTOR 

METRICS 

ACRONYM* 

CORRECTNESS 

COMPLETENESS  CHECKLIST 

CP.l 

PROCEDURE  CONSISTENCY  MEASURE 

CS.l 

DATA  CONSISTENCY  MEASURE 

CS.2 

DESIGN  STRUCTURE  MEASURE 

SI.1 

STRUCTURED  LANGUAGE  OR  PREPROCESSOR 

SI.2 

DATA  AND  CONTROL  FLOW  COMPLEXITY 

MEASURE 

SI.3 

CODING  SIMPLICITY  MEASURE 

SI.4 

SCOPE  OF  FUNCTION  MEASURE 

SP.l 

CROSS  REFERENCE 

TR.l 

RELIABILITY 

ERROR  TOLERANCE/CONTROL  CHECKLISTS 

AM.l 

IMPROPER  INPUT  DATA  CHECKLIST 

AM.2 

COMPUTATIONAL  FAILURES  CHECKLIST 

AM.3 

HARDWARE  FAULTS  CHECKLIST 

AM.4 

DEVICE  ERROR  CHECKLIST 

AM.5 

COMMUNICATION  ERRORS  CHECKLIST 

AM.6 

NODE/COMMUNICATIONS  FAILURES 

AM.7 

ACCURACY  CHECKLIST 

AY.l 

PROCEDURE  CONSISTENCY  MEASURE 

CS.l 

DATA  CONSISTENCY  MEASURE 

CS.2 

DESIGN  STRUCTURE  MEASURE 

SI.1 

STRUCTURED  LANGUAGE  OR  PREPROCESSOR 

SI.2 

DATA  AND  CONTROL  FLOW  COMPLEXITY 

MEASURE 

SI.3 

CODING  SIMPLICITY  MEASURE 

SI.4 

EFFICIENCY 

PERFORMANCE  REQUIREMENTS 

EF.l 

ITERATIVE  PROCESSING  EFFICIENCY  MEASURE 

EF.2 

DATA  USAGE  EFFICIENCY  MEASURE 

EF.3 

STORAGE  EFFICIENCY  MEASURE 

EF.4 

♦Acronym  references  relate  to  definitions  in  Appendix  C 
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Table  2.4-3  (Continued) 


QUALITY 

FACTOR 

METRICS 

ACRONYM* 

INTEGRITY 

ACCESS  CONTROL  CHECKLIST 

SA.l 

ACCESS  AUDIT  CHECKLIST 

SA.2 

SYSTEM/DATA  INDEPENDENCE  CHECKLIST 

VR.l 

USABILITY 

USER  INPUT  INTERFACE  MEASURE 

CM.l 

USER  OUTPUT  INTERFACE  MEASURE 

CM.2 

OPERABILITY  CHECKLIST 

OP.l 

TRAINING  CHECKLIST 

TN.l 

SYSTEM/DATA  INDEPENDENCE  CHECKLIST 

VR.l 

MODULE  TESTING  MEASURE 

VS.I 

INTEGRATION  TESTING  MEASURE 

VS.2 

SYSTEM  TESTING  MEASURE 

VS.  3 

SURVIVABILITY 

ERROR  TOLERANCE/CONTROL  CHECKLIST 

AM.I 

IMPROPER  INPUT  DATA  CHECKLIST 

AM. 2 

COMPUTATIONAL  FAILURES  CHECKLIST 

AM. 3 

HARDWARE  FAULTS  CHECKLIST 

AM.4 

DEVICE  ERRORS  CHECKLIST 

AM.5 

COMMUNICATION  ERRORS  CHECKLIST 

AM. 6 

NODE/COMMUNICATIONS  FAILURES  CHECKLIST 

AM. 7 

INTERFACE  COMPLEXITY  MEASURE 

AU.l 

SELF-SUFFICIENCY  CHECKLIST 

AU.2 

DESIGN  STRUCTURE  CHECKLIST 

DL1 

MODULAR  IMPLEMENTATION  MEASURE 

MO.2 

MODULAR  DESIGN  MEASURE 

MO.3 

RESTRUCTURE  CHECKLIST 

RE.  I 

MAINTAINABILITY 

HALSTEAD'S  MEASURE 

CO.l 

PROCEDURE  CONSISTENCY  MEASURE 

CS.l 

♦Acronym  references  relate  to  definitions  in  Appendix  C 
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Table  2.4-3  (Continued) 


QUALITY 

FACTOR 


ACRONYM* 


MAINTAINABILITY 


(continued) 


DATA  CONSISTENCY  MEASURE 
MODULAR  IMPLEMENTATION  MEASURE 


MODULAR  DESIGN  MEASURE 


QUANTITY  OF  COMMENTS 
EFFECTIVENESS  OF  COMMENTS  MEASURE 
DESCRIPTIVENESS  OF  LANGUAGE  MEASURE 


DESIGN  STRUCTURE  MEASURE 
STRUCTURED  LANGUAGE  OR  PREPROCESSOR 
DATA  AND  CONTROL  FLOW  COMPLEXITY 


MEASURE 


CODING  SIMPLICITY  MEASURE 
MODULE  TESTING  MEASURE 
INTEGRATION  TESTING  MEASURE 
SYSTEM  TESTING  MEASURE 


VERIFIABILITY 


MODULAR  IMPLEMENTATION  MEASURE 


MODULAR  DESIGN  MEASURE 


QUANTITY' OF  COMMENTS 
EFFECTIVENESS  OF  COMMENTS  MEASURE 
UES'.  .  TIVENESS  OF  IMPLEMENTATION 


LANGUAGE  MEASURE 


DESIGN  STRUCTURE  MEASURE 
STRUCTURED  LANGUAGE  OR  PREPROCESSOR 
DATA  CONTROL  FLOW  COMPLEXITY 


MEASURE 


CODING  SIMPLICITY  MEASURE 
SCOPE  OF  FUNCTION  MEASURE 
MODULE  TESTING  MEASURE 


♦Acronym  references  relate  to  definitions  in  Appendix  C 
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Table  2.4-3  (Continued) 


QUALITY 

FACTOR 

METRICS 

VERIFIABILITY 

INTEGRATION  TESTING  MEASURE 

VS.2 

(continued) 

SYSTEM  TESTING  MEASURE 

VS.3 

FLEXIBILITY 

MODULE  REFERENCE  BY  OTHER  MODULES 

GE.1 

IMPLEMENTATION  FOR  GENERALITY  CHECKLIST 

GE.2 

MODULAR  IMPLEMENTATION  MEASURE 

MO.2 

MODULAR  DESIGN  MEASURE 

MO,3 

QUANTITY  OF  COMMENTS 

SD.l 

EFFECTIVENESS  OF  COMMENTS  MEASURE 

SD.2 

DESCRIPTIVENESS  OF  LANGUAGE  MEASURE 

SD.3 

DESIGN  STRUCTURE  MEASURE 

SI.1 

STRUCTURED  LANGUAGE  OR  PREPROCESSOR 

DATA  AND  CONTROL  FLOW  COMPLEXITY 

SI.2 

MEASURE 

SI.3 

CODING  SIMPLICITY  MEASURE 

SI.4 

PORTABILITY 

SOFTWARE  SYSTEM  INDEPENDENCE  MEASURE 

ID.l 

MACHINE  INDEPENDENCE  MEASURE 

ID.2 

MODULAR  IMPLEMENTATION  MEASURE 

MO.2 

MODULAR  DESIGN  MEASURE 

MO.3 

QUANTITY  OF  COMMENTS 

SD.l 

EFFECTIVENESS  OF  COMMENTS  MEASURE 

SD.2 

DESCRIPTIVENESS  OF  LANGUAGE  MEASURE 

SD.3 

REUSABILITY 

DATA  BASE  SYSTEM  INDEPENDENCE 

AI.1 

DATA  STRUCTURE 

|gj§ 

ARCHITECTURE  STANDARDIZATION 

KpSal 

MICROCODE  INDEPENDENCE 

ALGORITHM 

ACCESS  NO-CONTROL 

mm, 

*  Acronym  references  relate  to  definitions  in  Appendix  C 
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Table  2.4-3  (Continued) 


QUALITY 

FACTOR _ ^ 

METRICS 

\CRONYM^ 

REUSABILITY 

WELL-STRUCTURED  DOCUMENTATION 

DA.2 

(continued) 

SELECTIVE  USABILITY 

DA.  3 

FUNCTION  SPECIFICITY 

FS.1 

FUNCTION  COMMONALITY 

FS.2 

FUNCTION  COMPLETENESS 

FS.3 

MODULE  REFERENCE  BY  OTHER  MODULES 

GE.1 

IMPLEMENTATION  FOR  GENERALITY  CHECKLIST 

GE.2 

SOFTWARE  SYSTEM  INDEPENDENCE 

ID.l 

MACHINE  INDEPENDENCE 

.0.2 

MODULAR  IMPLEMENTATION  MEASURE 

MO.2 

MODULAR  DESIGN  MEASURE 

MO.3 

INTERFACE  COMPLEXITY 

SC.1 

PROGRAM  FLOW  COMPLEXITY 

SC.2 

APPLICATION  FUNCTIONAL  COMPLEXITY 

SC.3 

COMMUNICATION  COMPLEXITY 

SC.4 

STRUCTURE  CLARITY 

SC.3 

QUANTITY  OF  COMMENTS 

SD.1 

EFFECTIVENESS  OF  COMMENTS  MEASURE 

SD.2 

DESCRIPTIVENESS  OF  LANGUAGE  MEASURE 

SD.3 

DESIGN  STRUCTURE  MEASURE 

SI.1 

STRUCTURED  LANGUAGE  OR  PREPROCESSOR 

DATA  AND  CONTROL  FLOW  COMPLEXITY 

SL2 

MEASURE 

SI.  3 

CODING  SIMPLICITY  MEASURE 

SI.4 

INTEROPERABI- 

DATA  STORAGE  EXPANSION  MEASURE 

LITY 

COMPUTATIONAL  EXTENSIBILITY  MEASURE 

CHANNEL  EXTENSIBILITY  MEASURE 

DESIGN  EXTENSIBILITY  CHECKLIST 

AG.4 

♦Acronym  references  relate  to  definitions  in  Appendix  C 
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Table  2.4-3  (Continued) 


♦Acronym  references  relate  to  definitions  in  Appendix  C 
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2.4.2 


Example  of  Metrics 


Using  the  example  of  paragraph  2.2.2,  the  quality  ratings  would  be  specified  as  follows. 

3a.  Specific  Quality  Factor  Ratings 

Ratings  for  two  of  the  five  important  quality  factors  can  be  established  usinfc 
Table  2.4-1. 

Reliability  .99  Require  less  than  one  error  per  100  lines  of  code  to  be 

detected  during  formal  testing. 

Maintainability  .8  Require  less  than  or  equal  to  2  man  days  as  an  average 

level  of  maintenance  for  correcting  an  error. 


These  ratings  can  also  be  established  at  each  measurement  period  (see  Table  3.1-1) 
during  the  software  development  process  as  follows: 


QUALITY 

MEASUREMENT  PERIODS 

Mi 

mm 

CDR 

IMPL 

ACCEPT 

FACTOR 

Reliability 

.8 

.8 

.9 

.9 

.99 

Maintainability 

.7 

.7 

.8 

.8 

.8 

The  progressively  better  scores  are  required  because  there  is  more  detailed 
information  in  the  later  phases  of  the  development  to  which  to  apply  the  metrics 
and  more  confidence  in  the  metrics'  indication  of  quality.  This  is  analagous  to  the 
concept  of  reliability  growth.  For  other  quality  factors  see  step  3b. 

3b.  Identify  Specific  Metrics  to  be  Applied 

The  metrics  to  be  applied  to  assess  the  level  of  each  important  quality  factor 
are  chosen  from  Table  2.4-3.  A  subset  is  shown  in  Table  2.4-4. 


Table  2.4*4  Software  Metric  to  Factor  Relationship-Subset 
|  QUALITY  FACTOR 


Rel- 

Main- 

Cor- 

Usa- 

labl- 

tain  - 

rect- 

bil  - 

lity 

abU  - 
ity 

ness 

ity 

METRIC 

Accuracy  Checklist 

X 

Error  Tolerance  Checklist 

• 

X 

* 

• 

Complexity  Measure 

X 

X 

Coding  Simplicity  Measure 

X 

X 

Modular  Implementation  Measure 
• 

X 

• 

Quantity  of  Comments 

X 

Effectiveness  of  Comments 

• 

X 

* 

• 

Cross  Reference  Checklist 

X 

Completeness  Checklist 

Halstead's  Measure 

X 

X 

Data  Consistency  Measure 
• 

X 

X 

• 

User  Input  Interface  Measure 

X 

Communications  Commonality 
Data  Commonality  Checklist 
• 

• 

• 

Documentation  for  Other  Systems 

Int  - 
erop- 
era- 
bil  - 
ity 


X 


xxx 


3c.  Specify  Threshold  Value* 

The  following  threshold  values  are  established  based  on  past  experience  and  to 
provide  a  goal  for  the  quality  factors  that  were  not  given  ratings.  They  were 
derived  by  determining  the  average  scores  of  past  applications  of  the  metrics. 


Cross  Reference  Checklist  .  9 

Completeness  Checklist  1 . 0 

Halstead's  Measure  .  9 

Data  Consistency  Measure  .  6 

T raining  Checklist  .  7  5 

User  Input  Interface  Measure  .75 

User  Output  Interface  Measure  .75 

Communications  Commonality  .  8 

Data  Commonality  Checklist  .  8 


2.5  EVALUATION  OF  DEVET  OPMENT  PLAN 

In  an  acquisition  environment  the  initial  benefits  of  utilizing  the  quality  metrics  concepts 
are  realized  in  the  source  selection  process.  The  acquisition  office  should  include  the 
quality  goals  established  as  software  requirements  in  the  Request  for  Proposal.  The 
software  attributes  should  also  be  identified  as  required  characteristics  in  the  software 
and  the  metrics  established  as  the  vehicles  for  assessing  their  existence.  The  bidders 
should  be  required  to  describe  how  they  plan  to  provide  those  characteristics  in  the 
software.  This  discussion  should  be  provided  in  the  portion  of  the  proposal  that  describes 
their  development  plan. 

The  description  of  the  bidders  approach  for  including  the  required  attributes  in  the 
software  not  only  forces  acknowledgement  of  these  additional  requirements  but  also 
provides  additional  information  with  which  to  evaluate  the  bidders  during  source  selec¬ 
tion. 


SECTION  3 
APPLYING  METRICS 


3.1  WHEN  TO  TAKE  MEASUREMENTS 

The  software  quality  metrics  are  oriented  toward  the  availability  of  information  about  the 
software  system  as  it  progresses  in  its  development.  In  the  early  phases  of  the 
development,  the  metrics  are  applied  to  the  documentation  produced  to  describe  the 
concepts  of  the  system  and  its  design.  In  the  later  phases  the  metrics  are  oriented  not 
only  to  documentation  but  also  to  the  source  code  that  is  available. 

Thus,  the  application  of  the  metrics  logically  follows  the  phased  development  of  software. 
The  first  application  of  the  metric  is  at  the  end  of  the  requirements  analysis  phase.  The 
next  application  is  during  design.  If  the  design  phase  has  been  decomposed  into  a 
preliminary  dosign  phase  and  a  detailed  design  phase,  the  metrics  should  be  applied  at  the 
end  of  each  of  those  phases.  During  implementation,  i.e.,  coding,  the  metrics  oriented 
toward  the  source  code  should  be  applied  periodically  to  assess  the  quality  growth 
exhibited  as  the  code  evolves.  The  timing  of  the  application  of  the  metrics  is  shown  in 
Figure  3.1-1.  The  application  of  the  metrics  can  be  done  during  or  just  prior  to  formal 
customer  reviews  (as  shown  in  Figure  3.1-1)  or  during  equivalent  activities  conducted  by 
the  development  personnel. 

In  the  case  of  reusable  software,  metrics  may  already  exist  from  being  applied  during  a 
previous  project.  Other  metrics  may  change  when  re-evaluated  later  in  the  life-cycle, 
e.g.,  during  maintenance.  Maintainability,  reliability  and  expandability  factors  may  be  re¬ 
evaluated  as  maintenance  and  upgrade  activities  occurred  for  fielded  systems. 
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3.2 


SOURCES  OF  QUALITY  INFORMATION 


A  typical  minimum  set  of  documents  and  source  code  are  shown  in  Figure  3.2-1.  These 
documents  plus  the  source  code  are  the  sources  of  the  metrics  information  used  to  derive 
the  quality  ratings. 


REQUIREMENTS 

ANALYSIS 


PRELIM 

INARY 

DESIGN 


DETAILED 

DESIGN 


PROGRAMMING 

AND 

CHECKOUT 


TEST 

AND 

INTEGRATION 
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APPLICATION  OF  THE  METRICS 


Application  of  the  metrics  can  be  accomplished  by  using:  the  metric  worksheet  contained 
in  Appendix  A  for  gathering  data,  the  metric  tables  in  Appendix  B  to  translate  the 
measurements  into  metric  scores  and  the  data  in  Appendix  C  for  definitions  and 
interpretations  of  individual  metrics. 

The  metric  worksheets  are  organized  as  follows.  In  the  header  portion  of  the  worksheet  is 
the  information  which  (1)  identifies  the  phase  during  which  the  worksheet  is  initially  used 
and  the  level  (system  or  module)  to  which  the  worksheet  applies,  (2)  identifies  the  system 
and  the  module  to  which  the  worksheet  has  been  applied,  and  (3)  identifies  the  date  and 
the  inspector  who  took  the  measurements.  The  remaining  portion  of  each  worksheet 
contains  the  measurements  to  be  taken  and  questions  to  be  answered.  These  measure¬ 
ments  and  questions  are  organized  by  quality  factors  identified  in  parentheses.  Each 
logical  group  of  measurements  and  questions  have  a  group  identifier  and  group  number. 
Each  question  contains  a  reference  to  the  applicable  metric. 

When  applying  the  measurements,  only  those  measurements  and  questions  that  relate  to 
the  quality  factors  chosen  as  quality  goals  should  be  applied.  A  complete  metric 
worksheet  correlation  matrix  is  shown  in  Table  3.3.1.  The  metric  worksheet  correlation 
matrix  provides  a  quality  factor  to  metric  relationship.  It  also  provides  an  individual 
metric  to  metric  worksheet  relationship. 

Metric  Worksheet  #1  and  //2  contain  system  level  metrics  and  are  applied  at  the  system  or 
major  subsystem  (CPCI)  level  to  the  System  Requirements  Specification,  the  Preliminary 
Design  Specification,  the  User's  Manual,  and  the  Test  documentation.  Metric  Worksheets 
//3  and  #4  contain  module  level  metrics  and  are  applied  to  each  module's  design  (Detailed 
Design  Specification)  and  implementation  (source  code). 

The  metric  tables  in  Appendix  B  are  utilized  to  translate  the  raw  data  from  the  metric 
worksheets  into  individual  metric  scores.  The  metric  tables  in  Appendix  B  are  listed 
alphabetically  by  quality  criteria.  The  metric  tables  are  arranged  as  follows.  In  the 
header  portion  of  the  table  is  a  reference  to  the  quality  criteria  and  the  quality  factors. 
The  body  of  the  table  contains  the  instructions  for  computing  individual  metric  scores 
with  a  reference  to  the  metric  worksheet  that  the  raw  data  may  be  obtained  from. 
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COMCCTNESS 


KUTION 


MTtCttTIOI 


Definitions  and  interpretations  of  the  individual  measurements  contained  in  the  work¬ 
sheets  are  found  in  Appendix  C. 


As  shown  in  Figure  3.3-1,  the  worksheets  may  be  applied  several  times  during  the 
development.  For  example,  Metric  Worksheet  #3,  which  is  applied  for  each  module  to  the 
detailed  design  document  during  design,  is  also  applied  to  the  detailed  design  document 
after  it  has  been  updated  to  reflect  the  actual  implementation.  The  worksheet  does  not 
have  to  be  totally  reapplied  for  each  successive  application.  It  should  only  involve 
updates  to  reflect  the  changes  made  to  the  system  since  the  previous  application  of  the 
worksheet.  The  successive  applications  of  any  worksheet  should  require  considerably  less 
effort  than  the  original  application. 


Worksheet 

Number 


1 


2 


3 


4 


Requirements 

— 

Preliminary 

Detailed 

Test  and 

Analysis 

Design 

Design 

Implementation 

Integration 

Requirements 

Spec 


• 

Preliminary  ■  Preliminary 

Design  Spec  \  Design  Spec 

■ 

a 

Users  Manual  •  Users  Manual 

(Draft)  •  (Draft) 

-  M  ■  . 

Detail 
Design  Spec 

Test  Plans 
<Sc  Procedures 


Detail 
Design  Spec 

Test  Plans 
<5c  Procedures 


Test 

Results 


Users  Manual 

(Final)  s 

......a 


Source  Code 

Detail  Design 
Spec  (Updated) 


1st  Application 
Reapplication 


Figure  3.3-1  Application  of  the  Metric  Worksheets 
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3.4 


TECHNIQUES  FOR  APPLYING  METRICS 


Section  1.5  identified  organizational  approaches  for  utilizing  the  quality  metric  concepts 
during  software  development.  These  approaches  included  both  acquisition  environments 
and  internal  development  environments.  The  purpose  of  this  section  is  to  describe,  at  a 
lower  level,  how  the  metrics  would  be  applied  in  either  case. 

The  first  technique  for  applying  the  metrics  is  by  formal  inspection.  The  formal 
inspection  is  performed  by  personnel  of  an  organization  independent  of  the  development 
organization  (the  acquisition  office,  an  independent  quality  assurance  group,  or  an 
independent  IV&V  contractor).  The  metric  worksheets  are  applied  to  delivered  products 
at  scheduled  times  and  the  results  are  formally  reported. 

The  second  technique  is  to  utilize  the  worksheets  during  structured  design  and  code 
walkthroughs  held  by  the  development  team.  A  specific  participant  of  the  walkthrough 
can  be  designated  to  apply  the  worksheets  and  report  any  deficiencies  during  the 
walkthrough,  or  a  quality  assurance  person  can  participate  in  the  walkthroughs  to  take  the 
measurements  of  the  design  or  code. 

The  last  technique  is  for  the  development  team  to  utilize  the  worksheets  as  guidelines, 
self-evaluations  or  in  a  peer  review  mode  to  evaluate  or  enhance  the  quality  of  the 
products  they  produce. 
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SECTION  4 

ASSESSING  THE  QUALITY  OF  THE  SOFTWARE  PRODUCT 

4.1  INTRODUCTION 

The  benefits  of  applying  the  software  quality  metrics  are  realized  when  the  information 
gained  from  their  application  is  analyzed.  The  analyses  that  can  be  done  are  described  in 
the  subsequent  paragraphs.  There  are  three  levels  at  whicn  analyses  can  be  performed. 
These  levels  are  related  to  the  level  of  detail  to  which  the  evaluating  organization  wishes 
to  go  in  order  to  arrive  at  a  quality  assessment. 

4.2  INSPECTOR'S  ASSESSMENT 

The  first  level  at  which  an  assessment  can  be  made  relies  on  the  discipline  and 
consistency  introduced  by  the  application  of  the  worksheets.  An  inspector,  using  the 
worksheets,  asks  the  same  q>jestions  and  takes  the  same  counts  for  each  module's  source 
code  or  design  document,  etc.  that  is  reviewed.  Eased  on  this  consistent  evaluation,  a 
subjective  comparison  of  products  can  be  made. 

la.  Document  Inspector's  Assessment 

The  last  section  in  each  worksheet  is  for  the  inspector  to  make  comments  on 
the  quality  observed  while  applying  the  worksheet.  Comments  should  indicate 
an  overall  assessment  as  well  as  point  out  particular  problem  areas  such  as 
lack  of  comments,  inefficiencies  in  implementation,  or  overly  complex  control 
flow. 

lb.  Compile  Assessments  for  System  Review 

By  compiling  all  of  the  inspector's  assessments  on  the  various  documents  and 
source  code  available  at  any  time  during  the  development,  deficiencies  can  be 
identified. 
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4.3 


SENSITIVITY  ANALYSIS 


The  second  level  of  detail  utilizes  experience  gained  through  the  application  of  metrics 
and  the  accumulation  of  historical  information  to  take  advantage  of  the  quantitative 
nature  of  the  metrics.  The  values  of  the  measurements  are  used  as  indicators  for 
evaluation  of  the  progress  toward  the  high  quality  goals  or  requirements. 

At  appropriate  times  during  a  large-scale  development,  the  application  of  the  worksheets 
allows  calculation  of  the  metrics.  The  correspondence  of  the  worksheets  to  the  metrics  is 
shown  in  Appendix  B.  The  results  of  these  calculations  is  a  matrix  of  measurements.  The 
metrics  that  have  been  established  to  date  are  at  two  levels,  system  level  and  module 
level.  The  approach  described  is  for  the  module  level  metrics  however  it  is  applicable  to 
both  levels. 

A  n  by  k  matrix  of  measurements  results  from  the  application  of  the  metrics  to  the 
existing  products  (e.g.,  at  design,  the  products  might  include  review  material,  design 
specifications,  test  plans',  etc.)  where  there  are  k  modules  and  n  module  level  measure¬ 
ments  applicable  at  this  particular  time. 


This  matrix  represents  a  profile  of  all  the  modules  in  the  system  with  respect  to  a  number 
of  characteristics  measured  by  the  metrics.  The  analyses  that  can  be  performed  are 
described  in  the  following  steps: 

2a.  Assess  Variation  of  Measurements 

Each  row  in  the  above  matrix  represents  how  each  module  in  the  system 
scored  with  respect  to  a  particular  metric.  By  summing  all  the  values  and 
calculating  the  average  and  standard  deviation  for  that  metric,  each  individual 
module's  score  can  then  be  compared  with  the  average.  Those  modules  that 
score  more  than  one  standard  deviation  below  the  average  should  be  reported 
for  further  examination.  These  calculations  are  illustrated  below: 


mu  mi2  .  .  .  .  mik 
m21 


•  • 
mnl  mnk 
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V 


for  metric  i{  Average  Score  =  Aj  =  £  Mjj/k-# 

J-l 


/k  \ 

Standard  Deviation  =  °j  =  (D(Mjj**Ai)2/k  1 

\j“l  / 

Report  Module  j  if  Mij<Ai  -  Oi 
Assess  Low  System  Scores 

In  examining  a  particular  measure  across  all  modules,  consistently  low  scores 
may  exist.  It  may  be  that  a  design  or  implementation  technique  used  widely 
by  the  development  team  was  the  cause.  This  situation  indicates  the  need  for 
a  new  standard  or  stricter  enforcement  of  existing  standards  to  improve  the 
overall  development  effort. 


2c.  Assess  Scores  Against  Thresholds 

As  experience  is  gained  with  the  metrics  and  data  is  accumulated,  threshold 
values,  or  industry  acceptable  limits,  may  be  established.  The  scores,  for  each 
module  for  a  particular  metric  should  be  compared  with  the  established 
threshold.  A  simple  example  is  the  percent  of  comments  per  iine  of  source 
code.  Certainly  code  which  exhibits  only  one  or  two  percent  measurements 
for  this  metric  would  be  identified  for  corrective  action.  It  may  be  that  ten 
percent  is  a  minimum  acceptable  level.  Another  example  is  the  complexity 
measure.  A  specific  value  of  the  complexity  measure  greater  than  some 
chosen  value  should  be  reported  for  corrective  action,  e 


Report  Module  j  if  Mjj  <  T[ 

Where  Tj  =  threshold  value  specified  for  metric  i. 
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4.4 


USE  OF  NORMALIZATION  FUNCTION  TO  ASSESS  QUALITY 


The  last  level  of  assessing  quality  is  using  the  normalization  functions  to  predict  the 
quality  in  quantitative  terms.  The  normalization  functions  are  utilized  in  the  following 
manner. 

At  a  particular  time  in  the  development  process  there  is  an  associated  matrix  of 
coefficients  which  represent  the  results  of  linear  multivariate  regression  analyses  against 
empirical  data  (past  software  developments).  These  coefficients,  when  multiplied  by  the 
measurement  matrix  results  in  an  evaluation  (prediction)  of  the  quality  of  the  product 
based  on  the  development  to  date.  This  coefficient  matrix,  shown  below,  has  n  columns 
for  the  coefficients  of  the  various  metrics  and  13  rows  for  the  13  quality  factors. 


To  evaluate  the  current  degree  or  level  of  a  particular  quality  factor,  i,  for  a  module,  j, 
the  particular  column  in  the  measurement  matrix  is  multiplied  by  the  row  in  the 
coefficient  matrix.  The  resultant  value: 


cl,l  cl,2  . .  .  cl,n 


C13,l  c13,n 


r.  .  c-  ,  m.  . 
LJ  =  hi  i,J 


+  ci,2  m2,j ' 


.  .  +  c-  rri  . 
i,n  m,j 


is  the  current  predicted  rating  of  that  module,  j  for  the  quality  factor,  i.  This  predicted 
rating  is  then  compared  to  the  previously  established  rating  to  determine  if  the  quality  is 
at  least  as  sufficient  as  required.  The  coefficient  matrix  should  be  relatively  sparse 
(many  C..  =  0).  Only  subsets  of  the  entire  set  of  metrics  applicable  at  any  one  time 
relates  to  the  criteria  of  any  particular  quality  factor. 


Multiplying  the  complete  measurement  matrix  by  the  coefficient  matrix  results  in  a 
ratings  matrix.  This  matrix  contains  the  current  predicted  ratings  of  each  module  for 
each  quality  factor.  Each  module  then  can  be  compared  with  the  preset  rating  for  each 
quality  factor. 
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CM=RJJ>= 


Pi, l  rt,2  .  .  .  rl,3 


1^13,1  r13,k  J 

This  represents  the  most  formal  approach  to  evaluating  the  quality  of  a  product  utilizing 
the  software  quality  metrics.  Because  the  coefficient  matrix  has  been  developed  only  for 
a  limited  sample  in  a  particular  environment,  it  is  neither  generally  applicable  nor  has 
statistical  confidence  in  its  value  been  achieved. 

To  use  the  normalization  functions  that  currently  exist,  the  following  steps  should  be 
performed. 


3a.  Apply  Normalization  Function 

Table  4.4-1  contains  the  normalization  functions  that  currently  exist.  If  any 
of  the  quality  factors  identified  in  that  table  have  been  specified  as  a 
requirement,  then  the  metrics  identified  in  the  table  should  be  substituted  into 
the  equation  and  the  predicted  rating  calculated.  Normalization  functions 
which  include  several  metrics  can  be  used  if  available,  otherwise  functions  for 
individual  metrics  should  be  used.  This  predicted  rating  should  be  compared 
with  the  specified  rating. 

To  illustrate  the  procedure,  the  normalization  function  that  has  been  deve¬ 
loped  for  the  factor  Flexibility  will  be  used.  The  normalization  function, 
applicable  during  the  design  phase,  relates  measures  of  modular  implementa¬ 
tion  (MO.2)  to  the  flexibility  of  the  software.  The  predicted  rating  of 
flexibility  is  in  terms  of  the  average  time  to  implement  a  change  in 
specifications.  The  normalization  function  is  shown  in  Figure  4.4-1.  The 
measurements  associated  with  the  modular  implementation  metric  are  taken 
from  design  documents.  The  measurements  involve  identiiving  if  input,  output 
and  processing  functions  are  mixed  in  the  same  module,  if  application  and 
machine-dependent  functions  are  mixed  in  the  same  module  and  if  processing 
is  data  volume  limited.  As  an  example,  assume  the  measurements  were 
applied  during  the  design  phase  and  a  value  of  0.65  was  measured.  Inserting 
this  value  in  the  normalization  function  results  in  3  predicted  rating  for 
flexibility  of  .33  (.51  x  .65)  as  identified  by  point  A  in  Figure  4.4-1,  If  the 
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Acquisition  Manager  had  specified  a  rating  of  0.2,  which  is  identified  by  point 
B,  he  has  an  indication  that  the  software  development  is  progressing  well  with 
respect  to  this  desired  quality. 

An  organization  using  this  manual  is  encouraged  to  establish  these  functions  in 
its  specific  environment  by  following  the  procedures  described  in  (MCCA77), 
(MCCA80),  or  Volume  1  of  this  report. 


Table  4.4-1  Normalization  Functions 


RELIABILITY  (DESIGN) 


MULTIVARIATE  -18  M.u  .  ♦  .19  Me,  , 

FUNCTION 

INDIVIDUAL  .34  MAM  { 

FUNCTIONS  .34  M$I  3 

AM.l  Error  Tolerance/Control  Checklist 
SI.3  Data  and  Control  Flow  Complexity 
Measure 

RELIABILITY  (IMPLEMENTATION) 

MULTIVARIATE  .48  MAM  {  +  .14  M$I  j 

FUNCTION 

INDIVIDUAL  .57  MAM  . 

FUNCTIONS  .58  M$I  * 

•53  MSI.‘3 
.53  M$i>4 

AM.l  Error  Tolerance/' Control  Checklist 
SI.l  Design  Structure  Measure 

51.3  Data  and  Control  Flow 

Complexity  Measure 

51.4  Coding  Simplicity  Measure 

MAINTAINABILITY  (DESIGN) 

INDIVIDUAL  .57  MSI 

FUNCTIONS 

.53  MSI<1 

SI.3  Data  and  Control  Flow 

Complexity  Measure 

SI.l  Design  Structure  Measure 
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Table  4.4-1  (Continued) 


MAINTAINABILITY  (IMPLEMENTATION) 

MULTIVARIATE 

FUNCTION 

-.2+. 61  MSI  3+  .14Mmq  2+.33sd>2 

INDIVIDUAL 

SI.3  Data  and  Control  Flow 

FUNCTIONS 

Complexity  Measure 

2.1  MSI3 

MO.2  Modular  Implementation  Measure 

•7l  MSD.2 

SD.2  Effectiveness  of  Comments 

•6  MSD.3 

Measure 

*5  MSI.l 

SD.3  Descriptivenes>s  of  Language 

*4  MSI.4 

Measure 

SI.l  Design  Structure  Measure 

SI.4  Coding  Simplicity  Measure 

FLEXIBILITY  (DESIGN) 

INDIVIDUAL 

•51  MM0.2 

MO.2  Modular  Implementation  Measure 

FUNCTIONS 

•56  MGE.2 

GE.2  Implementation  for  Generality 
Checklist 

FLEXIBILITY  (IMPLEMENTATION) 

MULTIVARIATE 

FUNCTION 

,22MM0.2  +  *44MGE.2  +  -09MSD.3 

INDIVIDUAL 

FUNCTIONS 

*6  MMO,2 

MO.2  Modular  Implementation  Measure 

•72MGE.2 

GE.2  Implementation  for  Generality 

*59  MSD.2 

Checklist 

-56  MSD.3 

SD.2  Effectiveness  of  Comments 

Measure 

5D.3  Descriptiveness  of  Language 

Measure 

Table  4.4-1  (Continued) 


PORTABILITY  (IMPLEMENTATION) 

MULTIVARIATE 

FUNCTION 

-1.7  -t-  •l9MsD.l  +  *76MSD.2  +  2,5MSD.3  +  *64MID.2 

INDIVIDUAL 

FUNCTIONS 

l.°7  MSI 

SD.l  Quantity  of  Comments 

1.1  mid>1 

SD.2  Effectiveness  of  Comments 

1,5  MSD.2 

Measure 

SD.3  Descriptiveness  of  Language 

Measure 

!D.2  Machine  Independence  Measure 
cI.l  Design  Structure  Measure 

REUSABILITY 

MULTIVARIATE 

♦  13  +  *29  |  +  .O8M5J  ^ 

FUNCTIONS 

.10  +  -08MSD.l  +  *l9MSD.3  +  •07MSi.3 

.11  +  -04MpS1  +  •06M5d>)  +  >16Msd.3  t  *07MSI.3 

.il  +  .03Mfs<i  +  *0'*Msc>4  +  .06Msd<1  +  *14Msd,3 
+  *06MSI.3 

INDIVIDUAL 

FUNCTIONS 

.22  .12  *  MpS>l 

FS.i  Function  Specificity 

.03  +  .28  *  mge,2 

GE.2  Implementation  for  Generality 

.14  +  .I7  *  MjD<2 

Checklist 

.20  ♦  .19  *  Mmo>2 

ID. 2  Machine  Independence  Measure 

.18  +  .21  *  MsC.l 

MO.2  Modular  Implementation  Measure 

.22  ♦  .14  -  Msc<2 

SC.i  Interface  Complexity 

.14  +  .24  *  M5c.4 

SC.2  Program  Flow  Complexity 

.23  ♦  .16  *  MSD1 

SC.4  Communication  Complexity 

.01  +  .36  *  MSD  3 

SD.l  Quantity  of  Comments 

.10  +  .37  *  MSI  1 

SD.3  Descriptiveness  of  Language 

.26  ♦  .13  *  M$i;3 

Measure 

Table  4.4*1  (Continued) 


-.14  ♦  .56  *  Msia 


SI.  1  Design  Structure  Measure 

51.3  Data  and  Control  Flow  Complexity 

Measure 

51.4  Coding  Simplicity  Measure 
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3b.  Calculate  Confidence  in  Quality  Assessment 

Using  statistical  techniques  a  level  of  confidence  can  be  calculated.  The 
calculation  is  based  on  the  standard  error  of  estimate  for  the  normalization 
function  and  can  be  derived  from  a  normal  curve  table  found  in  most  statistics 
texts.  An  example  of  the  derivation  process  is  shown  in  Figure  4.4-2  for  the 
situation  described  above.  Here  it  is  shewn  that  the  Acquisition  Manager  has 
an  86  percent  level  of  confidence  that  the  flexibility  of  the  system  will  be 
better  than  the  specified  rating. 


MEAN  =  .33 


I 


MEAN  =  .33  (PREDICTED  RATING) 

STANDARD  DEVIATION  =  .12  (STANDARD  ERROR  OF  ESTIMATE) 
LEVEL  OF  CONFIDENCE  =  Pr  [x  >  .2]  =  .86  (SHADED  AREA) 


Figure  4.4-2  Determination  of  Level  of  Confidence 
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REPORTING  ASSESSMENT  RESULTS 


Each  of  the  preceding  step*  described  in  this  section  are  easily  automated.  If  the  metrics 
are  applied  automatically  then  the  metric  data  is  available  in  machine  readable  form.  If 
the  worksheets  are  applied  manually,  then  the  data  can  be  entered  into  a  file,  used  to 
calculate  the  metric,  and  formatted  into  the  measurement  matrix  format.  The  automa¬ 
tion  of  the  analyses  involve  simple  matrix  manipulations.  The  results  of  the  analyses 
should  be  reported  at  various  levels  of  detaii.  The  formats  of  the  reports  are  left  to  the 
discretion  of  the  implementing  organization.  The  content  of  the  reports  to  the  different 
managers  is  recommended  in  the  following  paragraphs. 

I  a.  Report  to  the  Acquisition  Manager/Development  Manager 

The  report  content  to  the  Acquisition  Manager  and  the  Development  Manager 
should  provide  summary  information  about  the  progress  of  the  development 
toward  the  quality  goals  identified  at  the  beginning  of  the  project. 

For  example  if  ratings  were  specified  for  several  quality  factors,  the  current 
predicted  ratings  should  be  reported. 


PREDICTED  RATING 

QUALITY  GOALS  BASED  ON  DESIGN  DOCUMENT 

RELIABILITY  .9  .8 

MAINTAINABILITY  .8  .95 

If  specific  ratings  were  not  identified  but  the  important  qualities  were  identified,  a  report 
might  describe  the  percentage  of  modules  that  currently  are  judged  to  be  below  the 
average  quality  (as  a  result  of  the  sensitivity  analysis)  or  that  are  below  a  specified 
threshold  value  (as  a  result  of  the  threshold  analysis).  These  statistics  provide  a  progress 
status  report  to  the  manager.  Further  progress  status  is  indicated  by  reporting  the  quality 
growth  of  the  system  or  of  individual  modules.  The  quality  growth  is  depicted  by 
reporting  the  scores  achieved  during  the  various  phases  of  development.  Ultimately  the 
ratings  should  progressively  score  higher  than  those  reported  during  the  requirements 
phase.  This  progress  is  based  on  the  identification  of  problems  in  the  early  phases  which 
can  then  be  corrected. 

lb.  Reports  to  Quality  Assurance  Manager 

In  addition  to  the  summary  quality  progress  reports  described  in  la,  the  quality 
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assurance  manager  and  his  staff  will  want  detailed  metric  reports.  These 
reports  will  provide  all  of  the  results  of  the  Analyses  described  in  4.2,  4.3,  and 
4.4,  and  perhaps  provide  the  measurement  matrix  itself  for  examinations.  In 
addition  to  the  detailed  reports,  the  quality  assurance  manager  should  be 
provided  with  reports  on  the  status  of  the  application  of  the  metrics 
themselves  by  the  quality  assurance  staff.  These  status  reports  will  provide 
information  on  the  total  number  of  modules  and  the  number  which  inspectors 
have  analyzed. 

lc.  Reports  to  the  Development  Team 

The  development  team  should  be  provided  detailed  information  on  an  excep¬ 
tion  basis.  This  information  is  derived  from  the  analyses.  Examples  of  the 
information  would  be  quality  problems  that  have  been  identified,  which 
characteristics  or  measurements  of  the  software  products  are  poor,  and  which 
modules  have  been  identified  as  requiring  rework.  These  exception  reports 
should  contain  the  details  of  why  the  assessment  revealed  them  as  potential 
problems.  It  is  based  on  this  information  that  corrective  actions  will  be  taken. 
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APPENDIX  A 
METRIC  WORKSHEETS 


Appendix  A  contains  the  metric  worksheets  which  are  used  to  gather  metric  data 
during  the  software  development  phases.  There  are  four  worksheets,  organized  by 
applicable  phase: 

Worksheet  1  -  Requirements  Analysis 
Worksheet  2  -  Preliminary  Design 
Worksheet  3  -  Detailed  Design 
Worksheet  4  -  Source  Code 

A  summary  of  the  worksheets  is  shown  on  the  next  page.  Each  worksheet  is  divided 
into  sections  of  related  questions  to  ease  the  data  gathering  task.  The  applicable 
metric  element  is  referenced  by  acronym  at  the  end  of  each  worksheet  question. 
Appendix  B,  Metric  Tables,  lists  the  formula  to  be  used  in  calculating  values  for 
metrics  and  metric  elements. 

The  contents  of  this  appendix  are  based  on  the  results  of  this  contract,  "Quality 
Metrics  for  Distributed  Systems",  F30602-80-C-0330  and  the  results  of  contract 
F30602-80-C-0265,  "Software  Interoperability  and  Reusability".  This  appendix  includes 
a  refinement  and  reorganization  of  worksheet  information  initially  defined  in 
RADC-TR-77-369  and  RADC-TR-80-109, 
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METRIC  WORKSHEETS  SUMMARY 


I  M  impttto*  iCommtnU 


METRIC  WORKSHEET.  1 

iYSTEMi 

DATE: 

IBvl  -M?i  t  J  -vy  i  ■-  f M  i 

NAME: 

INSPECTOR: 

1.1  STRUCTURE  (RELIABILITY,  MAINTAINABILITY,  VERIFIABILITY,  FLEXIBILITY,  REUSABILIT 
EXPANDABILITY,  SURVIVABILITY,  PORTABILITY,  INTEROPERABILITY,  CORRECTNESS) 


1.  Is  an  organization  of  the  system/network  provided  which  identifies  all  software  func¬ 
tions  and  functional  interfaces  in  the  system?  OI.l(l) 

2.  Number  of  major  functions.  SI.  1(2) 

3.  Are  there  no  duplicate  functions?  SI.  1(2) 

4.  Is  there  a  definitive  statement  of  the  requirements  for  the  distibution  of  information 
within  the  data  base?  01.1(3) 

5.  Is  there  an  organization  of  the  data  base  provided  which  identifies  the  types  of  system- 
level  information  and  the  information  flow  within  the  system?  DLK2) 

6.  Is  there  a  definitive  statement  of  requirements  for  code  to  be  written  according  to  a  pro 
gramming  standard?  SI.4(13) 

7.  Is  there  a  definitive  statement  of  requirements  for  processes,  functions,  and  modules  to 
have  loose  coupling?  MO,3(l) 

S.  Is  there  a  definitive  statement  of  requirements  for  processes,  functions,  and  modules  to 
have  high  cohesion?  MO,3(2) 


1.2  TOLERANCE  (RELIABILITY,  SURVIVABILITY) 


1 . 

Has  an  error  analysis  been  performed  and  budgeted  to  functions?  AY.  1(1) 

H 

■ 

2. 

Are  there  definitive  statements  of  the  accuracy  requirements  for  inputs,  outputs, 
ptocessing,  ana  constants?  AY.  1(2) 

H 

N 

3. 

Are  there  definitive  statements  of  the  error  tolerance  of  input  data?  AM.2(1) 

a 

El 

4. 

Are  there  definitive  statements  of  the  requirements  for  recovery  from  computa¬ 
tional  failures?  AM.3(1) 

R 

n 

3. 

Is  there  a  definitive  statement  of  the  requirement  for  recovery  from  hardware 
faults?  AM.l'(l) 

R 

N 

6. 

Is  there  a  definitive  statement  of  the  requirements  for  recovery  from  device 
errors?  AM.5(1) 

Y 

R 

7. 

Are  there  definitive  statements  of  the  requirements  for  recovery  from  communication 

errors?  AM.60) 

_ 

_L 

1 

1 

N 

□ZJ 

II 

■1 

H 

H 

fl 

N 

Y 

N 

R 

N 

R 

9 

-3 


S.  Are  there  definitive  statements  of  the  requirements  for  system  recovery  from  node 
or  communication  failures?  AM.7(1) 


1.3  PERFORMANCE  (EFFICIENCY) 


l.  Have  performance  requirements  ana  limitations  (flow  time  for  process,  including  execu¬ 
tion  and  communication;  storage)  been  specified  for  the  functions  to  be  performed?  EF.  1(1) 


1.*  COMPLETENESS  (CORRECTNESS) 


1.  Is  there  a  matrix  relating  itemized  requirements  to  major  functions  which  implement 
those  requirements?  TR.  1(1) 

2.  Number  of  major  functions  identified  (equivalent  to  CPOI).  CP.l 

3.  Are  requirements  itemized  so  that  the  various  functions  to  be  performed,  their  inputs 
and  outputs,  are  clearly  delineated?  CP. Hi) 

4.  Number  of  major  dalta  references.  CP.  1(2) 

3.  How  many  of  these  data  references  arc- not  defined?  CP.  1(2) 

6.  How  many  defined  functions  are  not  used?  CP.U3) 

7.  How  many  referenced  functions  are  not  defined?  CP. 1(4) 

8.  How  many  data  references  are  not  used?  CF  1(2) 

9.  How  many  referenced  data  references  are  not  defined’’  CP.  1(6) 

10.  is  the  flow  of  processing  and  all’decision  points  in  that  flow  described.  CP. 1(5) 

11.  How  many  problem  reports  related  to  the  requirements  v.)  been  recorded?  CP. 1(7) 

12.  How  many  of  those  problem  reports  have  been  closed  (r  solved)?  CP.  1(7) 


1.5  FUNCTIONAL  SCOPE  (REUSABILITY) 


Is  the  function  constructed  ir.  such  a  way  to  encourage  its  use  elsewhere  either  in 
part  or  in  total?  FS.2U) 

Are  the  input  quantities  well  defined?  F$.2(2) 

Are  the  output  well  defined  and  easy  to  interpret?  FS,2(4) 

Do  the  functions  performed  satisfy  one  of  the  specified  requirements?  F5.2(3) 
Number  of  function  requirements  satisfied  by  the  reusable  software?  F5.311) 
Total  number  of  requirements?  F5.3(l) 


SYSTEMt 


INSPECTOR: 


14  CHANGEABILITY  (INTEROPERABILITY,  EXPANDABILITY) 


U  there  a  definitive  statement  of  requirements  tor  spare  storage  capacity  (memory  and 
auxiliary  storage)?  AG,  1(2,3) 

Is  there  a  definitive  statement  of  requirements  for  spare  processing  capacity?  AG.2(3) 

Is  there  a  definitive  statement  of  requirements  for  spare  I/O  and  communication  channel 
capacity?  AG.3ii,2) 

Is  there  a  definitive  statement  of  requirements  for  interface  compatibility  *  mong  all  the 
processors,  communication  links,  memory  devices,  wtd  peripherals?  AG.4(1) 

Is  there  a  specific  requirement  for  providing  performance/price  information  for  enhance¬ 
ment  traoes?  AG.4(2) 

Do  specifications  idertify  new  technology  tradeoff  areas  for  software?  AG, 4(3) 

Do  software  specifications  include  requirements  for  the  criteria  of  the  quality  factor 
expandability?  AG.4(4) 


SYSTEM  INTERFACES  (INTEROPERABILITY,  S UR YW ABILITY) 

Is  there  a  definitive  statement  of  the  requirements  for  communication  with  other 
systems?  CL. 1(1) 

Are  there  specific  requirements  lor  network  process  control?  CL.  1(3) 

Are  there  specific  requirements. for  user  session  control?  CL.  1(6) 

Are  the-*  specific  requirements  for  a  communication  routing  strategy?  CL.K7) 

Is  there  a  definitive  statement  of  the  requirements  for  standard  data  representa¬ 
tions  for  communication  with  other  systems?  CL. 2(1) 

Are  processes  and  functions  separated  as  logical  "wholes"  to  minimize  interface  complex¬ 
ity?  AU.  ') 

Are  tiere  specific  requirements  for  each  CPU/system  to  have  a  separate  power  source? 
AU.2<1) 

Are  there  specific  requirements  for  each  software  scheduling  unit  to  test  its  own  opera¬ 
tion,  communication  .'inks,  memories,  and  peripherals?  AU.2(3) 

Are  there  specific  requirements  for  the  software  system  to  include  a  word  processing 
capability?  AU.2(3) 

Are  there  specific  requirements  for  network  communication  capabilities  in  the  event 
of  failure  of  a  node  or  communication  link?  RE.  HI) 


Are  there  specific  requirements  for  a  node  to  rejoin  the  network  when  it  has  been  recov¬ 
ered?  RE.  1(4) 

Is  there  a  definitive  statement  of  the  operating  procedures  to  be  used  with  this  system? 
CL.IU5) 

Is  there  a  low  dependency  on  handshaking  time  between  systems?  CL.Kl  1) 

How  many  systems  must  respond  correctly  to  successfully  complete  handshaking?  CL.HIO) 
Are  there  no  timing  dependencies  on  the  system  communication  response  time  that  effect 
system  performance  requirements?  CL.  1(1 2) 

Are  there  no  timing  dependencies  on  the  freshness  of  data  that  effect  system  per'ormance 
requirements?  CL,  1(1 4) 


14  DATA  BASE  (SURVIVABILITY,  USABILITY,  INTEGRITY,  EXPANDABILITY,  CORRECTNES 
RELIABILITY,  MAINTAINABILITY) 


1.  Is  there  a  definitive  statement  of  the  requirements  for  maintaining  data  base  integrity 
under  anomalous  conditions?  RE.  1(2) 

2.  Are  there  specific  requirements  for  file/library  accessibility  from  each  node?  DI.  1(4) 

3.  Are  there  specific  requirements  for  a  virtual  storage  structure?  VR.l(l) 

4.  Is  thei  ‘  a  definitive  statement  of  the  requirements  for  establishing  and  verifying  data 
base  consistency  and  concurrency  at  each  node  which  hosts  a  data  base  partition?  CS.2(4) 


1.9  HUMAN  INTERFACE  (USABILITY,  INTEROPERABILITY) 


Are  all  steps  in  the  operation  described  (operations  concept)?  OP. 1(1) 

Are  all  error  conditions  to  be  reported  to  operator/user  identified  and  the 
responses  described?  OP, 1(2) 

Is  there  a  statement  of  the  requirement  for  the  capability  to  interrupt  operation, 
obtain  operational  status,  save,  modify,  and  continue  processing?  OP.  1(3) 

Is  there  a  statement  of  the  requirement  for  the  capability  to  obtain  network  resource 
status?  OP.  1(9) 

Is  there  a  definitive  statement  of  requirements  for  optional  input  media?  CM.  1(6) 

Is  there  a  definitive  statement  of  requirements  for  optional  output  media?  CM.2(7) 

Is  there  c  definitive  statement  of  requirements  for  selective  output  control?  CM.2(1) 
Is  there  a  definitive  statement  of  requirements  for  selection  of  different  nodes  for 
different  types  of  processing  or  for  different  types  of  information  retrieval?  OP. 1(10) 
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METR’C  WORKSHEET  l 

SYSTEM: 

REQUIREMENTS  ANALYSIS/SYSTEM  LEVEL 

NAME: 

DATE; 


1BSESCI2& 


Is  there  a  definitive  statement  of  requirements  for  establishing  standard  user  interlaces 


for  network  information  and  data  access?  CM.2(S) 

B 

N 

1.10 

COMMON  VOCABULARY  (INTEROPERABILITY) 

1. 

Do  both  projects  use  the  same  technical  vocabulary  with  identical  meanings?  CL.3(1) 

fl 

N 

l.ll 

DOCUMENTATION  (REUSABILITY,  INTEROPERABILITY) 

l. 

Is  there  no  access  control  to  the  software  document?  DA.  1(1) 

Y 

N 

2. 

Are  the  documents  clearly  and  simply  written?  DA.2G) 

D 

n 

3. 

Do  the  documents  contain  software  flow  charts  with  adequate  information  and  explana¬ 
tion?  DA.2(2) 

Y 

N 

4. 

Do  the  documents  have  hierarchical  structured  table  of  contents?  DA.2(3) 

a 

a 

3. 

Do  the  documents  have  index  system?  DA. 2(4) 

D 

rm 

6. 

Do  the  documents  have  separate  volumes  based  on  function?  DA. 2(3) 

a 

a 

7. 

Do  the  documents  have  functional  range  of  the  system?  DA. 2(6) 

a 

a 

3. 

Do  the  documents  describe  the  functions  performed?  DA. 2(7) 

a 

a 

9. 

Do  the  documents  describe  the  algorithm  used  and  limitations?  DA.2($) 

a 

a 

10. 

Do  the  documents  describe  the  relationship  between  functions?  DA, 2(9) 

a 

SB 

ti. 

Do  the  documents  contain  the  software  program  listing?  DA. 2(10) 

a 

m 

12. 

Do  the  programs  have  selective  computation/output  options?  DA. 3(1) 

a 

n 

13. 

Are  the  functions  performed  generally  associated  with  request  application?  DA.3(3) 

a 

a 

14. 

is  the  other  system  documentation  available  in  a  form  that  is  up-to-date,  complete  and 
clearly  organized?  SY.3G) 

Y 

N 

1.1/  SECURITY  (INTEGRITY) 


1. 

2. 

3. 


Is  there  a  definitive  statement  of  the  requirements  for  user  input/output  access  con¬ 
trols?  SA.l(l) 

is  there  a  definitive  statement  of  the  requirements  for  data  base  access  controls? 
SA.1(2) 

is  there  a  definitive  statement  of  the  requirements  for  memory  protection  across  task? 
SA.1(3) 


Y  N 


4.  b  there  a  definitive  statement  of  the  requirements  for  recording  and  reporting  access 
to  system?  SA.2(1) 

5.  b  there  a  definitive  statement  of  the  requirements  for  immediate  indication  of  access 
violation?  SA.2<2) 

6.  b  there  a  definitive  statement  of  the  requirements  for  network  access  controls? 
SA.1(4) 


1.14  INSPECTOR'^  COMMENTS 


Make  any  general  or  specific  comments  that  relate  to  the  quality  observed  while  applying  tr 
checklist. 


1.13  FUNCTIONAL  OVERLAP  (INTEROPERABILITY) 


l.  How  many  functions  are  duplicated  in  the  systems  that  are  to  interoperate?  FO.l(l) 

l.  How  many  of  these  duplicated  functions  will  be  deleted  in  one  or  the  other  system? 

FO.U2) 

3.  How  many  of  these  duplicated  function  pairs  will  require  to  be  synchronized?  FO.lO) 

4.  How  many  of  these  duplicated  function  pairs  will  require  redundancy  management  logic 

to  combine  them?  FO,l(4) 


2-1  STRUCTURE  (RELIABILITY,  MAINTAINABILITY,  VERIFIABILITY,  FLEXIBILITY,  REUSABILIT 
EXPANDABILITY,  SURVIVABILITY,  PORTABILITY,  INTEROPERABILITY,  INTEGRITY,  USABILITY, 
CORRECTNESS) 


1.  Is  an  organization  of  the  system  provided  which  identifies  all  functions  and  functional 
interfaces  in  the  system?  DI.  1(1) 

2.  Is  a  hierarchy  of  system  identifying  ail  modules  in  the  system  provided?  51.1(1) 

3.  Are  there  no  duplicate  functions  or  modules?  SL1(2) 

4.  Is  an  organ  ation  of  the  data  base  provided  which  identifies  all  functional  groupings 
of  data  and  data  flow  within  the  system?  DI.K2) 

5.  Are  there  provisions  for  selecting  alternate  processing  capabilities?  DI.1(J) 

6.  Are  critical  system  functions  distributed  over  redundant  elements  or  nodes?  DM (6) 

7.  Does  the  distribution  of  control  functions  ensure  network  operation/integrity  under  ano¬ 
malous  conditions?  DI.1(7) 

8.  Are  logical  structure  and  function  separated  in  the  design?  DI.l(S) 

9.  Are  physical  structure  and  iunction  separated  in  the  design?  Dl.l(9) 

10.  Number  of  nodes  that  ran  be  removed  and  still  have  each  node  able  to  communicate  with 
each  remaining  node.  01.1(10) 

11.  Do  processes  and  functions  have  loose  coupling?  MO,3(l) 

12.  What  is  the  cohesion  value  of  processes  and  functions?  MO.  3(2) 

13.  Can  each  user  utilize  the  system  as  though  it  were  dedicated  to  that  user?  VR.K4) 

14.  Is  the  user  presented  with  a  complete  logical  system  witho  ut  regard  to  physical  topology? 
VR.K3) 

13.  Do  module  descriptions  include  identification  of  module  interfaces?  SI.  1(9) 

16.  Has  a  programming  standard  been  developed?  51.1(8) 

17.  Number  of  modules  with  mixed  input/output  and  computational  functions?  SC.3(1) 

18.  (s  the  common  function  not  distributed  in  different  modules?  5C.3(4) 

19.  Does  the  module  not  perform  many  (related  but  different)  functions?  SC.3(5) 

20.  Number  of  modules  which  do  not  perform  single  function.  MO,2(S) 

21.  Are  the  modules  hierarchically  constructed?  M0.2(l) 


2.2  TOLERANCE  (RELIABILITY,  SURVIVABILITY) 


1.  Have  accuracy  requirements  been  budgeted  to  functions?  AY.  1(6) 


METRIC  WORKSHEET  2 

SYSTEM*. 

DATE?  . 

DESIGN/SYSTEM  LEVEL 

NAMEi 

INSPECTOR! 

2. 

Have  math  library  routines  to  be  used  been  checked  for  sufficiency  with  regards  to 
accuracy  requirements?  AY.  1(3) 

1 

N 

3. 

Is  concurrent  processing  centrally  controlled?  AM.  1(1) 

D 

o 

4. 

Is  parallel  processing  centrally  controlled? 

AM. 1(4) 

D 

Q 

5. 

How  many  error  conditions  are  reported  by  the  system?  AM.  1(2) 

6. 

How  many  of  those  errors  are  automatically  fixed  or  bypassed  and  processing  continues? 

AM.  1(2) 

■ 

7. 

How  many,  require  operator  intervention?  AM. 1(2) 

S. 

Are  there  provisions  for  recovery  from  hardware  faults?  AM.4(2) 

n 

n 

9. 

Are  there  provisions  for  recovery  from  device  errors?  AM. 3(2) 

n 

a 

10. 

Are  there  provisions  for  recovery  from  communication  errors?  AM. 6(2) 

D 

m 

11. 

Are  there  provisions  for  system  recovery  from  node  or  communication  failures?  AM.7(2) 

H 

D 

2.3 

OPTIMIZATION  (EFFICIENCY) 

Have  storage  requirements  and  limitations  been  allocated  to  functions?  EF.4(1) 

H 

H 

2. 

Are  virtual  storage  facilities  used?  EF.4(2) 

B 

m 

3. 

Is  dynamic  memory  management  used?  EF.4(5) 

B 

m 

4. 

Is  a  performance  optimizing  compiler  used?  EF.4(7) 

B 

m 

5. 

Have  Data  Base  or  files  been  organized  for  efficient  processing?  EF.3(l,5) 

B 

El 

6. 

Are  data  base  files/libraries  stored  at  only  one  node?  EF.4(8) 

B 

a 

7. 

Is  data  packing  used?  EF,2(5) 

B 

m 

S. 

Number  of  overlays  EF.2(4) 

9. 

Overlay  efficiency  -  memory  allocation  EF.2(4) 
max  overlay  size 

■ 

min  overlay  size 

10. 

Has  program  been  segmented  for  efficient  storage?  EF.4(4) 

D 

9 

11. 

Have  performance  requirements  and  limitations  been  allocated  to  functions?  EF.l(l) 

B 

11 

2.4 

COMPLETENESS  (CORRECTNESS) 

1. 

Is  there  a  matrix  relating  system  level  requirements  to  functions  which  implement  those 
requirements?  TR.l(l) 

1 

1 
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METRIC  WORKSHEET  2 

PKWmflTSMlSyBL 


SYSTEM;  DATE; 

NAME;  INSPECTOR; 


2.  How  many  major  functions  (CPCt’s)  are  identified?  CP.  1 

3.  Are  requirements  itemized  in  such  a  way  that  the  functions  to  be  performed,  their 
inputs  and  outputs  are  clearly  delineated?  CP.lO) 

4.  How  many  functions  identified  a.  e  not  defined?  CP. 1(4) 

5.  How  many  defined  functions  are  not  used?  CP.lO) 

6.  How  many  interfaces  between  functions  are  not  defined?  CP.  1(6) 

7.  Number  of  total  problem  reports  recorded?  CP.  1(7) 

8.  Number  of  those  reports  that  have  not  been  closed  (resolved)?  CP.  1(7) 

9.  Profile  of  problem  reports;  (number  of  following  types) 


Y 


N 


a. 

Computational 

h.  Routine/System 

P* 

Recurrent  errors 

b. 

Logic 

Interface 

q* 

Documentation 

c. 

Input/output 

i.  Tape  Processing 

r. 

Requirement  compliance 

d. 

Data  handling 

j.  User  interface 

s. 

Operator 

e. 

OS/System  Support 

k.  Data  base  interface 

t. 

Questions 

f. 

Configuration 

1.  User  requested 

u. 

Hardware 

8- 

Rout  me/Rou  tine 

changes 

V. 

Network  protocol 

Interface 

m.  Preset  data 

w. 

Communication  routing 

n.  Global  variable 
definition 


2.3 


1, 

2 

3 

4 
i 


REFERENCES  (REUSABILITY) 


Number  of  modules  with  database  system  reference.  AI.l(l) 

Number  of  modules  with  computer  architecture  reference.  AI.3U) 
Number  of  modules  are  not  in  standard  computer  architecture.  A1.3(2) 
Number  of  modules  used  microcode  instruction  statements.  A1.4(l) 
Number  of  modules  used  the  table  driven  algorithm.  Al.5(2) 


2.6  CHANGEABILITY  (FLEXIBILITY,  REUSABILITY,  EXPANDABILITY,  INTEROPERABILITY) 


1.  Percent  of  memory  capacity  uncommitted.  AG.  1(2) 

2.  Percent  of  auxiliary  storage  capacity  uncommitted.  AG.H3) 
3..  Percent  of  speed  capacity  uncommitted.  AG.2(3) 
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4.  Spare  1/0  channel  capacity.  AG. 3(1) 

3.  Spare  communication  channel  capacity.  AG. 3(2) 

6.  Are  processors,  communication  links,  memory  devices,  and  peripherals  compatible 
(o f  a  common  vendor  or  model)?  AG. 4(1) 

7.  Does  documentation  reveal  perfarmance/price  of  software/system  for  enhancement  trades? 
AG.4(2) 

8.  Do  specifications  identify  new  technology  tradeoff  areas  for  software?  AG.4(3) 

9.  Do  software  specifications  include  requirements  for  the  criteria  of  the  quality  factor 
expandability.  AG. 4(4) 

10.  Based  on  hierarchy  or  a  call/called  matrix,  how  many  modules  are  called  by  more  than 
one  module?  GE.l(l) 

11.  Number  of  modules.  GE.l(l) 


2.7  SYSTEM  INTERFACES  (INTEROPERABILITY ,  SURYIV ABILITY) 


1.  How  many  nodes  will  this  network/system  interface  with?  Cl _ 1(1) 

2.  Have  protocol  standards  been  established  for  network  process  control?  CL. 1(2) 

3.  Have  protocol  standards  been  established  for  user  session  control?  CL.  1(8) 

4.  Have  protocol  standards  been  established  for  communication  routing?  CL. 1(9) 

5.  Are  they  being  complied  with?  CL.1(2) 

6.  Number  of  modules  used  for  input  to  other  systems?  CL. 1(3) 

7.  Number  of  modules  used  for  output  to  other  systems?  CL.  1(4) 

8.  Has  a  standard  data  representation  been  established  or  translation  standards 
between  representations  been  established?  Are  they  being  compiled  with?  CL.2(2) 

9.  Number  of  modules  used  to  perform  translations?  CL.2(3) 

10.  Is  configuration  of  communication  links  such  that  failure  of  one  node/link  will  not 
disable  communication  among  other  nodes?  RE.  1(1) 

11.  Can  node  rejoin  the  network  when  it  has  been  recovered?  RE.1(4) 

12.  Is  data  replicated  at  two  or  more  distinct  nodes?  RE. 1(5) 

13.  Are  processes  and  functions  separated  as  logical  "wholes"  to  minimize  interface  complex¬ 

ity?  AU.  1(1) 

14.  Estimated  number  of  lines  of  interface  code.  AU.H2) 

13.  Estimated  number  of  interface  modules.  AU.K3) 
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NAME: 


INSPECTOR: 


16.  Estimated  time  engaged  in  communication.  AU.1(4) 

17.  Does  each  CPU/aystem  have  a  separate  power  source?  AU. 2(1) 

IS.  Does  each  scheduling  unit  test  its  own  operation,  communication  links,  memories,  and 
peripherals?  AU.2(2) 

19.  Does  the  software  system  include  a  worci-orocessing  capabili*y?  AU.2(3) 

20.  How  many  other  systems  will  this  system  interface  with?  CL. Hi 3) 


DATA  BASE  (RELIABILITY,  MAINTAINABILITY,  VERIFIABILITY,  FLEXIBILITY,  REUSABU.IT 
EXPANDABILITY,  USABILITY,  INTEGRITY,  SURVIVABILITY,  CORRECTNESS) 


1.  Number  of  unique  data  items  in  data  base  Sl.l(6) 

2.  Number  of  preset  data  items  SI.1(6) 

3.  Number  of  major  segments  (files)  in  data  base  S1.H7) 

4.  ts  the  data  base  structured  so  that  at  least  one  copy  of  s  file/library  resides  at  a  node 

which  is  accessible  to  ail  other  nodes?  DU(4) 

3.  Is  the  data  base  structured  so  that  users  need  not  care  about  changes  in  the  actual 
storage  structure  of  data?  VR.H2) 

6.  Are  there  provisions  for  maintaining  data  base  integrity  under  anomalous  conditions?  RE.1(3) 

7.  Can  users  manipulate  data  as  if  it  were  not  replicated  elsewhere  in  the  system?  VR.H3) 

8.  Have  procedures  been  established  for  verifying  data  base  consistency  and  concurrency 
at  each  node  which  hosts  a  data  base  partition?  .  C5.2(3) 

9.  Are  ail  data  centrally  controlled  and  symbolically  defined  and  referenced?  AI.2(3) 


2.9  HUMAN  INTERFACE  (USABILITY,  INTEROPERABILITY) 


1.  Are  all  steps  in  operation  described  including  alternative  flows?  OP. 1(1) 

2.  Number  of  operator  actions?  OP. 1(4) 

3.  Estimated  or  Actual  time  to  perform?  OP.  1(4) 

4.  Budgeted  time  for  complete  job?  OP.I  (4) 

3.  Are  job  set  up  and  tear  down  procedures  described?  OP.  1(3) 

6.  Is  a  hard  copy  of  operator  interactions  to  be  maintained?  OP.  1(6) 

7.  Number  of  operator  messages  and  responses?  OP.  1(7) 

8.  Number  of  different  formats?  OP.  1(7) 

9.  Are  all  error  conditions  and  responses  appropriately  described?  OP. 1(2) 

10.  Are  all  access  violations  and  responses  appropriately  described?  OP.  1(8) 


ISBBBBSRfflBJBi 


Does  the  capability  exist  lor  the  operator  to  interrupt,  obtain  status,  save, 
modify,  and  continue  processing?  OP.  1(3) 

Does  the  capability  exist  for  the  operator  to  obtain  network  resource  status.  OP.  1(9) 

Can  different  nodes  be  selected  for  different  types  of  processing  or  for  different 
types  of  information  retrieval?  OP.  1(10) 

Are  lesson  plans/training  materials  for  operators,  end  users,  and  maintainers 
provided?  TN.l(l) 

Are  realistic,  simulated  exercises  provided?  TN.K2) 

Are  help  and  diagnostic  information  available?  TN.1(3) 

Number  of  different  input  record  formats  CM. 1(2; 

Number  of  input  values  CM.  1(3) 

Number  of  default  values  CM. 1(1) 

Total  number  of  parameters  CM. 1(1) 

Number  of  self-identifying  input  values  CM.  1(3) 

Can  input  be  verified  by  user  prior  to  execution?  CM.  1(4) 

Is  input  terminated  by  explicitly  defined  by  logical  end  of  input?  CM.  1(3) 

Can  input  be  specified  from  different  media?  CM.  1(6) 

Are  there  selective  output  controls?  CM.2U) 

Do  outputs  have  unique  descriptive  user  oriented  labels?  CM. 2(2) 

Do  outputs  have  user  oriented  units?  CM. 2(3) 

Number  of  different  output  formats?  CM.2<4) 

Are  logical  groups  of  output  separated  for  user  examination?  CM.2(3) 

Are  relationships  between  error  messages  and  outputs  unambiguous?  CM.2(6) 

Are  there  provisions  for  directing  output  to  different  media?  CM. 2(7) 

Are  there  standards  governing  the  user  interlace  for  network  information  and  data 
access?  CM. 2(8) 

Are  the  standards  being  complied  with?  CM.2(3) 

Are  there  selectable  leveii  of  aid  and  guidance  for  users  of  different  degrees  of  expertise? 
TN.K4) 


2.10  TESTING  (USABILITY,  MAINTAINABILITY,  VER1FIABOJTYV- APPLY  TO  TEST  PLAN,  PROCEDURE 
RESULTS 


1.  Number  of  paths?  VS.Kl) 

2.  Number  of  paths  to  be  tested?  VS.l(l) 


METRIC  WORKSHEET  2 

SYSTEM:  |DATE: 

DESICN/SYSTEM  LEVEL _ 

NAME: _ INSPECTOR: _ 

3. 

Number  of  input  parameters7  VS.  1(2) 

4. 

Number  of  input  parameters  to  be  tested?  VS.  1(2) 

3. 

Number  of  interfaces?  VS.  2(1) 

6. 

Number  of  interfaces  to  be  tested?  VS.2(  1 ) 

7. 

Number  of  itemized  performance  requirements?  VS..?(2) 

8. 

Number  of  performance  requirements  to  be  verified?  VS.2(2) 

9. 

Number  of  modules?  VS.  3(1) 

10. 

Number  of  modules  to  be  exercised.  VS.30) 

11. 

Are  test  inputs  and  outputs  provided  in  summary  form?  VS.3(2) 

Y 

N 

2.11 

SYSTEM  COMPATIBILITY  (INTEROPERABILITY) 

l. 

Same  I/O  transmission  rate:  in  both  systems?  SY.l(l) 

Y 

N 

2. 

Same  communication  protocol  in  both  systems?  SY.1(2) 

Y 

N 

3. 

Same  message  content  in  both  systems?  SY.H3) 

Y 

N 

4. 

Same  message  structure  md  sequence  in  both  systems?  SY.1(4) 

Y 

N 

3. 

Is  data  in  both  systems  ir.  the  same  format  (ASCII,  EBCDIC,...)?  5Y.2U) 

Y 

N 

6. 

Same  data  base  structur*  in  both  systems? 

5Y.2(2) 

Y 

N 

7. 

Same  data  base  access  techniques  in  both  systems?  SY.2(3) 

Y 

N 

8. 

Same  source  language  n  both  systems?  SY.4(l) 

Y 

N 

9. 

Same  operating  system  in  both  systems?  SY.4(2) 

Y 

N 

10. 

Same  support  software  in  both  systems?  SY.4(3) 

Y 

N 

11. 

Same  word  length  in  both  systems?  SY,3(1) 

Y 

N 

12. 

Same  interrupt  structure  in  both  systems? 

SY.3(2) 

Y 

N 

13. 

Same  instruction  set  in  both  systems?  SY.3(3) 

71 

N 

2.12 

SECURITY  (INTEGRITY) 

1. 

Are  user  Inpul/Output  access  controls  provided?  SA.  1(1) 

Y 

N 

2. 

Are  Data  Base  sccess  controls  provided?  SA.K2) 

Y 

N 

3. 

Is  memory  protection  across  tasks  provided?  SA.  1(3) 

Y 

n" 

4. 

Are  there  provisions  for  recording  and  reporting  access  to  system?  5\.2(1) 

Y 

N 

3. 

Are  network  access  controls  provided?  SA.1(4) 

Y 

N 

6. 

Are  there  provisions  for  immediate  indication  of  access  violation?  SA.2(2) 

Y 

N 

V 
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M FTP  1C  WORKSHEET .2 

SYSTEM! 

DATE. 
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NAME:  .  .  . 

INSPECTOR. 

2*13  INSPECTOR'S  COMMENTS 

Make  any  general  or  specific  comments  about  the  quality  observed  while  applying  this  checklist* 
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IGN/MOOULE  LEVEL 


MODULE  NAME: 


3>2  TOLERANCE  (RELIABILITY,  SURVIVABILITY) 


1.  When  an  error  condition  is  detected,  is’ it  passed  to  calling  module?  AM.  1(3) 

2.  Have  numerical  techniques  being  used  in  algorithm  been  analyzed  with  regards  to  accuracy 
requirements?  AY. 1(4) 

3.  Are  .alues  of  inputs  range  tested?  AM. 2(2) 

4.  Are  conflicting  requests  and  illegal  combinations  identified  and  checked?  AM. 2(3) 
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5. 

6. 

7. 

8. 

9. 

10. 

n. 

12. 

13. 

14. 
13. 


3.3 


1. 

2. 


3. 


4. 


5. 


6. 


Is  there  a  check  to  see  if  all  necessary  data  is  available  before  processing  begins?  AM. 2(3) 
Is  all  input  checked,  reporting  aU  errors,  before  processing  begins?  AM.2(4) 

Are  loop  and  multiple  transfer  index  parameters  range  tested  before  use?  AM.3(2) 

Are  subscripts  range  tested  before  use?  AM. 3(3) 

Are  outputs  checked  for  reasonableness  before  processing  continues?  AM. 3(4) 

Are  checksums  computed  and  transmitted  with  all  messages?  AM. 6(3) 

Are  checksums  computed  and  compared  upon  message  reception?  AM.6(4) 

Are  the  number  of  transmission  retries  limited?  AM.6(5) 

Are  adjacent  nodes  checked  periodically  for  operational  status?  AM. 7(3) 

Are  there  alternate  strategies  for  message  routing?  AM. 7(4) 

Have  accuracy  requirements  been  budgeted  to  modules?  AY. 1(6) 


OPTIMIZATION  (EFFICIENCY) 


Are  specific  performance  requirements  (storage  and  routine)  allocated  to  this  module? 
EF.l(l) 

Which  category  does  processing  fall  in;  EF.2 

Real-time 

On-line 

Time-constrained 
Non-time  critical 

Hew  many  loops  have  non-loop  dependent  statements?  EF.2U) 

Is  bit/byte  packing/unpacking  performed  in  loops?  EF.2(3) 

Is  data  indexed  or  reference  efficiently?  EF .3(3) 

Is  performance  optimizing  compiler/assembly  language  used?  EF.2(2) 
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3.4  COMPLETENESS  (CORRECTNESS) 

1.  Is  there  a  matrix  relating  functional  requirements  to  the  module  which  implements 
those  requirements?  TR .  1  ( 1 ) 

2.  Can  you  clearly  distinguish  inputs,  outputs,  and  the  function  being  peformed?  CP. 1(1) 

3.  How  many  data  references  are  not  defined,  computed,  or  obtained  from  an  external 
source?  Cr.l(2) 

4.  Are  all  conditions  and  processing  defined  for  each  decision  point?  CP. 1(3) 

5.  How  many  problem  reports  have  been  recorded  for  this  module?  CP.  1(7) 


H 

i 

n 

ta 

■ 

n 

91 

■  1 

A- 18 


iii*n 


6.  Number  of  problem  reports  still  outstanding  CP. 1(7) 

7.  Profile  of  Problem  Reports:  (Number  of  Following  Types) 

a.  Computational 

h.  Routine/System  Inter- 

p.  Recurrent  Errors 

b.  Logic 

face 

q.  Documentation 

C-  Input/Output 

i.  Tape  Processing 

r.  Requirement  Compliance 

d.  Data  Handling 

j.  User  Interface 

s.  Operator 

e.  System/OS  Support 

k.  Data  Base  Interface 

t.  Questions 

f.  Configuration 

1.  User  Requested  Changes 

u.  Hardware 

g.  Routine/Routine  Inter- 

m.  Preset  Data 

v.  Network  Protocol 

face 

n.  Global  Variable  Definition 

w.  Communication  Routing 

REFERENCES  (MAINTAIN ABILITY ,  FLEXIBILITY,  VERIFIABILITY,  PORTABILITY,  REUSABIL1 
INTEROPERABILITY ,  EXPANDABILITY,  SURVIVABILITY) 


Number  of  references  to  system  library  routines,  utilities  or  other  system  provided  facilities 
ID.1U) 

Is  a  common,  standard  subset  of  programming  language  to  be  used?  ID.  1(2) 

Is  the  programming  language  available  in  other  machines?  ID.  2(1) 

Number  of  input/output  actions.  ID. 2(2) 

Number  of  calling  sequence  parameters  MO. 2(3) 

How  many  calling  sequence  parameters  are  control  variables?  MO.20) 

Is  input  passed  as  calling  sequence  parameters  MO. 2(4) 

Is  output  passed  back  to  calling  module?  M0.2(5) 

U  control  returned  to  calling  module?  MO,2(6) 

Is  temporary  storage  not  shared  with  other  modules?  MO. 2(7) 

Does  the  module  associate  with  database  system?  Al.l(l) 

Number  of  the  domains  in  system  Al.J(l) 

Number  of  the  domains  algorithm  works  for  in  system  AI,5(1) 

Is  the  algorithm  certification  available?  A1.J(3) 
is  the  algorithm  test  data  available?  A1.3(4) 


DESIGN/MODULE  LEV 
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3.9  FUNCTIONAL  CATEGORIZATION 


Categorize  {unction  performed  by  this  module  according  to  following: 

CONTROL  -  an  executive  module  whose  prime  i  unction  is  to  invoke  other  modules. 

INPUT/OUTPUT  -  a  module  whose  prime  function  is  to  communicate  data  between  the 
computer  and  either  the  user  or  another  computer. 

PRE/POSTPROCESSOR  -  a  module  whose  prime  function  is  to  prepare  data  for  or  after 
the  invocation  of  a  computation  or  data  management  module. 
ALGORITHM  -  a  module  whose  prime  function  is  computation. 

DATA  MANAGEMENT  -  a  module  whose  prime  function  is  to  control  the  flow  of  data 
within  the  computer. 

SYSTEM  -  a  module  whose  function  is  the  scheduling  of  system  resources  for  other  modules. 
COMMUNICATION  -  a  module  whose  prime  function  is  to  manage  messasge  routing  between  nodes. 
NETWORK  MANAGEMENT  -  a  module  whose  prime  function  is  to  monitor  and 

contrcl  network-level  resources. 

3,10  INSPECTOR'S  COMMENTS 


Make  any  specific  or  general  comments  about  the  quality  observed  while  applying  this  checklist. 


MODULE  NAME: 


INSPECTOR! 


4.1  STRUCTURE  (RELIABILITY,  MAINTAINABILITY,  VERIFIABILITY,  FLEXIBILITY,  PORT  ABUT* 
REUSABIUTY,  EXPANDABILITY,  CORRECTNESS) 


Number  ol  line*  excluding  comments  SI.<*(2) 

Number  of  declarative  statements  Sl.4(9) 

Number  ol  data  manipulation  statements  51.4(9) 

Number  of  statement  labels  (Do  not  count  format  statements  S1.4(6) 

Number  of  entrances  into  module  SI.  1  (5) 

Number  ol  exits  from  module  SI.  1(3) 

Maximum  nesting  level  SI.4(7) 

Number  of  decision  points  (IF,  WHILE,  REPEAT,  DO,  CASE)  SLMl) 

Number  of  sub-decision  points.  SI.  3(1) 

Number  of  conditional  branches  (computed  go  to)  SI.4(S) 

Number  of  unconditional  branches  (GOTO,  ESCAPE)  SL4(8) 

Number  of  loops  (WHILE,  DO)  51.4(3,4) 

Number  of  loops  with  jumps  out  ol  loop  Sl.4(3) 

Number  of  loop  indices  thac  are  modified  SI.4(4) 

Number  of  constructs  that  perform  module  modifications  (SWITCH,  ALTER)  SL4(5)  (Also 
see  4.3,  M0.2(2)) 

Number  of  negative  or  complicated  compound  boolean  expressions  51.4(2) 

Is  a  structured  language  used  SI.7.(  1 ) 

Is  flow  top  to  bottom  (are  there  no  backward  branching  GOTOs)  51.4(1) 

Is  code  written  according  to  a  programming  standard?  SI.4(13) 

Are  macros  and  subroutines  used  to  avoid  repeated  and  redundant  code?  SI.4(14) 

Number  of  data  items  used  to  specify  the  interface.  SC.i(l) 

Number  of  data  items  passed  implicitly  across  interface  via  comrfion  global  data  without 
adequate  comments.  SC.K2) 

Number  of  nesting  levels  in  interface.  SC.  1(3) 

Number  of  interface  data  items  with  negative  qualification.  SC.1(4) 

Number  of  data  items  passed  across  module  interface.  SC.  1(3) 

Dees  the  module  have  comments  about  the  common  control  blocks,  common  data  blocks 
and  global  variable  names  in  moduie  interface?  SC.  1(6) 

Does  the  module  modify  other  modules?  SC.  1(7) 

Number  of  possibL  unique  execution  paths.  5C,2(1) 


METRIC  WORKSHEET  4 

SYSTEM  NAMEi  _  .  . 

DATE-. 

_ SOURCE  COD£/MODULE_LEVE!. _ 

MODULE  NAME: 

■INSPECTOR; _ 

29. 

Number  of  IF  statement*.  SC. 2(2) 

30. 

Number  of  function  CALL*.  SC.2(3) 

31. 

Number  of  control  variables  used  to  direct  execution  path  selection.  SC. 2(4) 

32. 

Number  of  DO  groups.  5C.2(3) 

33. 

Does  the  module  have  code  comments  about  calling  what  modules  and  called  by  what 
modules?  SC.2(6) 

Y 

N 

34. 

Does  the  module  share  temporary  storage  with  other  modules?  SC.3(2) 

Y 

N 

3J. 

Does  the  module  have  mixed  database-management  and  storage-management  routines? 

SC. 3(3) 

Y 

N 

36. 

Average  number  of  formal  parameters  in  each  routine.  SC.4(1) 

37. 

Average  number  of  common  global  variable*  used  in  each  module.  SC.4(2) 

38. 

Number  of  global  variables  modified  by  one  routine  and  referenced  by  another  routines. 

5C.4(3) 

39. 

Does  the  module  connect  to  other  modules  with  functional  name?  SC.4(4) 

Y 

N 

40. 

Does  the  module  communicate  with  other  modules  by  passing  control  elements?  SC.4(5) 

Y 

N 

41. 

Number  of  machine  level  language  statements.  AI.3(3) 

42. 

Does  the  module  with  logical  processing  depend  on  data  Storage  specification  and  re¬ 
quirement?  A(.2(4) 

Y 

N 

43. 

Does  the  program  compute  the  same  vaiue  more  than  once?  SC.3(l) 

Y 

N 

44. 

Does  the  program  insert  a  statement  which  never  needs  to  be  executed?  SC.5(2) 

Y 

N 

43. 

Does  the  program  maintain  a  constant  meaning  for  each  variable?  SC.5(3) 

Y 

N 

46. 

Does  the  program  use  the  unnecessary  intermediate  variables?  SC.5(4F 

Y 

N 

TOLERANCE  (RELIABILITY,  SURVIVABILITY) 

1. 

Are  loop  and  multiple  transfer  index  parameters  range  tested  before  use? 

AM. 3(2) 

Y 

N 

2. 

Are  subscript  values  range  tested  before  use?  AM. 3(3) 

Y 

N 

3. 

Vhen  an  error  condition  occurs,  is  it  passed  to  the  calling  module?  AM.  1(3) 

Y 

N 

4. 

Are  the  results  of  a  computation  checked  before  outputting  or  before  processing  continues? 

AM. 3(4) 

Y 

N 
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M  OPTIMIZATION  (EFFICIENCY) 


1.  Number  o f  mix  mode  expressions?  EF.3(3) 

2.  How  many  variables  are  initialized  when  declared?  EF.3(2) 

3.  How  many  loops  ha/e  non-loop  dependent  statements  in  them?  EF.2(1) 

4.  Do  loops  have  bit/byte  packing/unpacking?  EF.2(5),  EF.4(6) 

5.  How  miny  compound  expressions  defined  more  than  once?  EF.2(3) 


M  CONCISENESS  (MAINTAINABILITY)  -  SEC  METRIC  EXPLANATIONS 


1.  Number  of  operators  CO. 1(1) 

2.  Number  of  unique  operators  CO.  1(1) 

3.  Number  of  Operands  CO.l(l) 

4.  Number  of  unique  operands  CO.l(l) 


1.  Number  of  calls  to  other  modules  M0.2U) 

2.  Number  of  references  to  system  library  routines,  utilities,  or  other  system  provided  functions 
ID. 1(1) 

3.  Number  of  calling  sequence  parameters  M0.2(3) 

4.  How  many  elements  in  calling  sequences  are  not  parameters?  M0.20) 

5.  How  many  of  the  calling  parameters  (input)  are  control  variables?  M0.2(3) 

6.  How  many  parameters  passed  to  or  from  other  modules  are  not  defined  in  this  module? 
MO,2<3) 

7.  Is  input  data  passed  as  parameter?  MO. 2(4) 

Is  output  data  passed  back  to  calling  module?  MO. 2(5) 

9.  Is  control  returned  to  calling  module?  MO. 2(6) 

10.  Number  of  lines  of  code?  M0.2(2) 


4.6  CHANGEABILITY  (FLEXIBILITY,  INTEROPERABILITY,  REUSABILITY,  EXPANDABILITY) 


1.  Is  module  table  driven?  AG. 2(2) 

2.  Are  there  any  limits  to  data  values  that  can  be  processed?  GE.2(4) 
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APPENDIX  B 
METRIC  TABLES 


Appendix  B  contains  the  metric  tables  which  are  used  for  calculating  values  for 
metrics  and  metric  elements.  The  tables  are  organized  alphabetically  by  quality 
criteria  name  and  numerically  by  metric  acronym.  A  summary  of  the  metric  tables 
and  a  correlation  to  metric  worksheets  are  shown  on  the  next  several  pages. 

Each  metric  table  identifies  the  quality  criteria,  the  metric,  and  the  metric  element 
and  references  the  applicable  quality  factors.  Formulas  are  stated,  where  appropriate, 
to  calculate  values  for  metric  elements  and  for  metrics.  Each  metric  element  is 
cross-referenced  to  the  software  development  phase  during  which  it  is  applicable  and 
to  the  appropriate  worksheet  and  worksheet  section(s)  (see  Appendix  A,  Metric  Work¬ 
sheets).  The  worksheet  cross-reference  is  by  a  decimal  number  scheme.  If,  for 
example,  1.2  is  called  out,  this  refers  to  Metric  Worksheet  l,  Section  2.  A  cross- 
reference  enclosed  in  parentheses  indicates  a  reapplication  of  the  metric  element 
during  a  subsequent  development  phase. 

Each  metric  in  the  tables  is  identified  by  a  type  code:  an  (a)  following  the  metric 
name  identifies  an  anomaly  detecting  metric,  and  a  (p)  identifies  a  predictive  metric. 
If  a  normalization  function  has  been  established  for  a  quality  factor  but  the  metric  is 
not  included,  it  is  because  the  metric  did  not  illustrate  sufficient  correlation  with  the 
operational  history.  In  lieu  of  inclusion  in  the  normalization  function,  some  metrics 
are  maintained  as  strictly  anomaly-detecting  metrics;  they  are  felt  to  identify  or  assist 
in  identification  of  problems  which  should  be  and  are  typically  corrected  immediately 
to  enhance  the  quality  of  the  product. 

The  contents  of  this  appendix  are  based  on  the  results  of  this  contract,  "Quality 
Metrics  for  Distributed  Systems",  F30602-8G-C-0330  and  the  results  of  contract 
F30603-80-C-02G5,  "Software  Interoperability  and  Reusability".  This  appendix  includes 
a  refinement  and  reorganization  of  metric  table  information  initially  defined  in 
RADC-TR-77 -369  and  RADC-TR-80-109. 
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METRIC  TABLES  SUMMARY 


CRITERIA 

METRICS 

ACCURACY 

AY.l 

ACCURACY  CHECKLIST 

ANOMALY  MANAGEMENT 

AM.  1 

ERROR  TOLERANCE/CONTROL  CHECK¬ 
LIST 

AM.2 

IMPROPER  INPUT  DATA  CHECKLIST 

AM.3 

COMPUTATIONAL  FAILURES  CHECKLIST 

AM. 4 

HARDWARE  FAULTS  CHECKLIST 

AM. 5 

DEVICE  ERRORS  CHECKLIST 

AM.6 

COMMUNICATION  ERRORS  CHECKLIST 

AM. 7 

NODE/COMMUNICATIONS  FAILURES 
CHECKLIST 

APPLICATION 

INDEPENDENCE 

AI.l 

DATA  BASE  SYSTEM  INDEPENDENCE 

AI.2 

DATA  STRUCTURE 

AI.3 

ARCHITECTURE  STANDARDIZATION 

AI.4 

MICROCODE  INDEPENDENCE 

AI.5 

ALGORITHM 

AUGMENTABILITY 

AG.  1 

DATA  STORAGE  EXPANSION  MEASURE 

AG.2 

COMPUTATION  EXTENSIBILITY  MEASURE 

AG.3 

CHANNEL  EXTENSIBILITY  MEASURE 

AG. 4 

DESIGN  EXTENSIBILITY  CHECKLIST 

AUTONOMY 

AU.l 

INTERFACE  COMPLEXITY  MEASURE 

AU.2 

SELF-SUFFICIENCY  CHECKLIST 

COMMONALITY 

CL.l 

COMMUNICATIONS  COMMONALITY 
CHECKLIST 

DATA  COMMONALITY  CHECKLIST 

- 

esB 

COMMON  VOCABULARY  CHECKLIST 

COMMUNICATIVENESS 

CM.  1 

USER  INPUT  INTERFACE  MEASURE 

CM. 2 

USER  OUTPUT  INTERFACE  MEASURE 

COM°LETENESS 

CP.l 

COMPLETENESS  CHECKLIST 

CONCISENESS 

CO.l 

HALSTEAD’S  MEASURE 

CONSISTENCY 

CS.l 

PROCEDURE  CONSISTENCY  MEASURE 

CS.2 

DATA  CONSISTENCY  MEASURE 

DISTRIBUTEDNESS 

DU 

DESIGN  STRUCTURE  CHECKLIST 

DOCUMENT  ACCESSIBILITY 

DA.l 

ACCESS  NO-CONTROL 

WELL-STRUCTURED  DOCUMENTATION 

m 

SELECTIVE  USABILITY 
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METRIC 


CRITERIA 

ACRONYM 

EFFECTIVENESS 

EF.l 

EF.2 

EF.3 

EF.4 

FUNCTIONAL  OVERLAP 

FO.l 

FUNCTIONAL  SCOPE 

FS.l 

FS.2 

FS.3 

GENERALITY 

GE.l 

GE.2 

INDEPENDENCE 

ID.l 

ID. 2 

MODULARITY 

OPERABILITY 

OP.  I 

RECONFIGURABILITY 

RE.  1 

SELF-DESCRIPTIVENESS 

SD.I 

SD.2 

SD.3 

SIMPLICITY 

su 

SI.2 

SI.3 

SI.4 

SPECIFICITY 

SP.I 

SYSTEM  ACCESSIBILITY 

SA.l 

SA.2 

SUMMARY 


METRICS 


PERFORMANCE  REQUIREMENTS 
ITERATIVE  PROCESSING  EFFICIENCY 
MEASURE 

DATA  USAGE  EFFICIENCY  MEASURE 
STORAGE  EFFICIENCY  MEASURE 


FUNCTIONAL  OVERLAP  MEASURE 


FUNCTION  SPECIFICITY 
FUNCTION  COMMONALITY 
FUNCTION  COMPLETENESS 


MODULE  REFERENCE  BY  OTHER  MOD¬ 
ULES 

IMPLEMENTATION  FOR  GENERALITY 
CHECKLIST 


SOFTWARE  SYSTEM  INDEPENDENCE  MEA¬ 
SURE 

MACHINE  INDEPENDENCE  MEASURE 


MODULAR  IMPLEMENTATION  MEASURE 
MODULAR  DESIGN  MEASURE 


OPERABILITY  CHECKLIST 


RESTRUCTURE  CHECKLIST 


QUANTITY  OF  COMMENTS 
EFFECTIVENESS  OF  COMMENTS  MEASURE 
DESCRIPTIVENESS  OF  LANGUAGE  MEAS¬ 
URE 


DESIGN  STRUCTURE  MEASURE 
STRUCTURED  LANGUAGE  OR  PRE¬ 
PROCESSOR 

DATA  AND  CONTROL  FLOW  COMPLEXITY 
MEASURE 

CODING  SIMPLICITY  MEASURE 


SCOPE  OF  FUNCTION  MEASURE 


ACCESS  CONTROL  CHECKLIST 
ACCESS  AUDIT  CHECKLIST 


METRIC  TABLES  SUMMARY 


CRITERIA 

ACRONYM 

METRICS 

SYSTEM  CLARITY 

SC.1 

INTERFACE  COMPLEXITY 

SC.2 

PROGRAM  FLOW  COMPLEXITY 

SC.3 

APPLICATION  FUNCTIONAL  COMPLEXITY 

SC.4 

COMMUNICATION  COMPLEXITY 

SC.5 

STRUCTURE  CLARITY 

SYSTEM  COMPATIBILITY 

SY.l 

COMMUNICATION  COMPATIBILITY 
CHECKLIST 

DATA  COMPATIBILITY  CHECKLIST 

HARDWARE  COMPATIBILITY  CHECKLIST 

SY.4 

SOFTWARE  COMPATIBILITY  CHECKLIST 

SY.j> 

DOCUMENTATION  FOR  OTHER  SYSTEM 

TRACEABILITY 

TR.l 

CROSS  REFERENCE 

TRAINING 

TN.l 

TRAINING  CHECKLIST 

VIRTUALITY 

VR.l 

SYSTEM/DATA  INDEPENDENCE  CHECK¬ 
LIST 

VISIBILITY 

Mgm 

MODULE  TESTING  MEASURE 

-  - 

INTEGRATION  TESTING  MEASURE 

wEam 

SYSTEM  TESTING  MEASURE 

(  )  •  UiWllutlJ*  of  Mocric  DurlA|  Sukat^uiQt  Fhua 
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CRITERIA:  ACCURACY  I  FACTOR(S):  RELIABILITY 


CRITERIA:  ANOMALY  MANAGEMENT  |  FACTOR(S):  RELIABILITY,  SURVIVABILITY 


CRITERIA:  ANOMALY  MANAGEMENT  |  FACTOR(S):  RELIABILITY,  SURVIVABILITY 


CRITERIA:  ANOMALY"  MANAGEMENT  IFACTOR(S):  RELIABILITY,  SURVIVABILITY 


CRITERIA:  APPLICATION  INDEPENDENCE  |  FACTOR(S):  REUSABILITY 


CRITERIA-.  APPLICATION  INDEPENDENCE  |  FACTOR(S):  REUSABILITY 


CRITERIA:  APPLICATION  INDEPENDENCE  FACTOR(S):  REUSABILITY 


CRITERIA:  APPLICATION  INDEPENDENCE  I  FACTOR(S):  REUSABILITY 


CRITERIA:  AUTONOMY  I  FACTOR(S):  SURVIVABILITY 


CRITERIA:  AUTONOMY  I  FACTOR(S):  SURVIVABILITY 


CRITERIA:  COMMONALITY  FACTOR(S):  INTEROPERABILITY 


CRITERIA:  COMMONALITY 


CRITERIA:  COMMUNICATIVENESS  1  FACTOR(S):  USABILITY,  INTEROPERABILITY 


CRITERIA:  COMPLETENESS  |  FACTOR(S):  CORRECTNESS 


t  applicable  elements 


CRITERIA:  CONSISTENCY  I  FACTOR(S):  CORRECTNESS,  RELIABILITY,  MAINTAINABILITY 


iw 


I 

I 


ft 

Kf 


I 
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FACTOR(S):  SURVIVABILITY  | 

Test  & 
Integration 

Value 

Yes/No 
1  or  0 

2.1 

2.1 

«Q 

oi 

2.1 

2.1 

2.1 

Implemen¬ 

tation 

Value 

Yes/No 
1  or  0 

Detail 

Design 

Value 

'5— 
2  o 

<J  0 
>■  - 

3.1 

Prel 

Design 

o 

Zo 

2.1 

2.1 

00 

CN 

<N 

2.1 

2.1 

2.1 

VI 

</» 

c 

a 

6 

V 

ta 

i 

02 

jj 

> 

Yes/No 
1  or  0 

- 

— * 

00 

.. 

CRITERIA:  DISTRIBUTEDNESS 

METRIC 

# 

DI.1  DESIGN  STRUCTURE  CHECKLIST  (a): 

(1)  Design  organization  identifies  all  functions 
and  interlaces 

(2)  Data  base  organization  identifies  all  data  and 
data  flow. 

(3)  Specific  requirements  for  information  distribution 
within  the  data  base. 

|  (4)  Provisions  for  file/library  access  from  other  nodes. 

(3)  Provisions  for  selecting  alternate  processing  capa¬ 
bilities. 

(6)  Critical  system  functions  distributed  over  redundant 
elements/nodes. 

(7)  Distribution  of  control  functions  ensures  network 
op-;  ation/integrity  under  anomalous  conditions. 

(8)  Logical  structure  and  function  separated  in  the 
design. 

(9)  Physical  structure  and  function  separated  in  the 
design. 

(10)  Number  of  nodes  that  can  be  removed  and  still 
have  each  node  able  to  communicate  with  each 
remaining  node  (Kleitman's  algorithm). 

«o 

c 

V 

s 

<u| 

3  a  1 

--  it  . 

S  £  1 

i 

-!! 

ri 

* 

o< 

l-l 

li 

U4 

u 

< 

> 

O 

S 

iU 

s 
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CRITERIA:  DOCUMENT  ACCESSIBILITY  I  FACTOR(S):  REUSABILITY 


CRITERIA:  DOCUMENT  ACCESSIBILi:  ••  1  FACTOR(S):  REUSABILITY 


CRITERIA:  EFFECTIVENESS  IFACTOR(S):  EFFICIENCY 


CRITERIA:  EFFECTIVENESS  |  FACTOR(S):  EFFICIENCY 


CRITERIA:  EFFECTIVENESS  FACTOR(S):  EFFICIENCY 


CRITERIA:  EFFECTIVENESS  |FACTOR(S):  EFFICIENCY 


CRITERIA:  EFFECTIVENESS  |  FACTOR(S):  EFFICIENCY 


CRITERIA:  FUNCTION  AL  OVERLAP  [  FACTOR(S):  INTEROPERABILITY 


CRITERIA:  FUNCTIONAL  SCOPE  FACTOR(S):  REUSABILITY 


CRITERIA:  INDEPENDENCE  !  FACTOR(S):  PORTABILITY,  REUSABILITY,  INTEROPERABILITY 


CRITERIA:  INDEPENDENCE  FACTOR(S):  PORTABILITY,  REUSABILITY,  INTEROPERABILITY 


CRITERIA:  OPERABILITY  FACTOR(S):  USABILITY 


CRITERIA:  RECONFIGURABILITY  |FACTOR<S):  SURVIVABILITY 


.s 

u  m 

j£ 

Value 

■ 

■ 

1 

If 

8  S 
>  _ 

(N 

■ 

a 

> 

t 

1.1 

Is 

Value 

■ 

■ 

|o 

£  5 

■ 

a 

n 

Value 

■ 

■ 

1 

1- 

£1 

■ 

a 

i 

c 

la 

Value 

■ 

■ 

1 

5® 

S  5 
>•  - 

m 

■ 

B 

i 
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METRIC 

r.E.l  RESTRUCTURE  CHECKLIST  (a): 

(1)  Configuration  of  communication  links  is  such  that 
failure  of  one  node/link  will  not  disable  communi¬ 
cation  among  other  nodes. 

(2)  Specific  requirements  for  maintaining  data  base 
integrity  under  anomalous  conditions 

(3)  Provisions  for  maintaining  data  base  integrity 
under  anomalous  conditions 
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(3)  Data  replicated  at  two  or  mere  distinct  nod*.;.. 

METRIC  VALUE  Tota*  sc0*-6  ,rom  applicable  elements 
§  applicable  elements 

PORTABILITY,  REUSABILITY 


ilicable  elements 


CRITERIA:  SELF-DESCRIPTIVENESS  FACTOR(S):  FLEXIBILITY,  MAINTAINABILITY,  VERIFIABILITY 

PORTABILITY,  REUSABILITY 


CRITERIA:  SELF-DESCRIPTIVENESS  FACTOR(S):  FLEXIBILITY,  MAINTAINABILITY,  VERIFIABILITY 

PORTABILITY,  REUSABILITY 


CRITERIA:  SIMPLICITY  FACTOR(S):  RELIABILITY,  MAINTAINABILITY,  VERIFIABILITY 

FLEXIBILITY,  REUSABILITY,  EXPANDABILITY, 
CORRECTNESS 


CRITERIA:  SIMPLICITY  FACTOR(S):  RELIABILITY,  MAINTAINABILITY,  VERIFIABILITY 

FLEXIBILITY,  REUSABILITY,  EXPANDABILITY, 
CORRECTNESS 


RITER1A:  SPECIFICITY  FACTOR(S):  CORRECTNESS,  VERIFIABILITY,  EXPANDABILITY 


CRITERIA:  SYSTEM  ACCESSIBILITY  FACTOR(S):  INTEGRITY 


CRITERIA:  SYSTEM  ACCESSIBILITY  |  FACTOR(S):  INTEGRITY 


CRITERIA:  SYSTEM  CLARITY  FACTOR(S):  REUSABILITY 


CRITERIA:  SYSTEM  CLARITY  jFACTOR(S):  REUSABILITY 


CRITERIA:  SYSTEM  CLARITY  1  FACTOR(S):  REUSABILITY 
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CRITERIA:  SYSTEM  CLARITY  EACTOR(S):  REUSABILITY 


CRITERIA:  SYSTEM  COMPATIBILITY  JFACTOR(S):  INTEROPERABILITY 


CRITERIA:  SYSTEM  COMPATIBILITY  |  FACTOR(S):  INTEROPERABILITY 


CRITERIA:  TRAINING  IPACTOR(S):  USABILITY 


CRITERIA:  VIRTUALITY  FACTOR(S):  INTEGRITY,  USABILITY,  EXPANDABILITY 


CRITERIA:  ViSIIMLITY  FACTOR(S):  VERIFIABILITY,  USABILITY,  MAINTAINABILITY 


APPENDIX  C 
METRIC  EXPLANATIONS 


Appendix  C  contains  a  detailed  explanation  of  each  metric  element.  The  explana¬ 
tions  are  organized  alphabetically  by  quality  criteria  and  numerically  by  metric 
acronym.  A  summary  of  the  metric  explanations  is  shown  on  the  next  several  pages. 
For  each  metric  element,  the  definition  (from  Appendix  B,  Metric  Tables)  is  stated, 
and  an  explanation  of  the  element  is  provided. 

The  contents  of  this  appendix  are  based  on  the  results  of  this  contract,  "Quality 
Metrics  for  Distributed  Systems",  F30602-80-C-0330  and  the  results  of  contract 
F30602-80-C-0265,  "Software  Interoperability  and  Reusability".  This  appendix  in¬ 
cludes  a  refinement  and  reorganization  of  metric  explanation  information  initially 
defined  in  RADC-TR-77-369  and  RADC-TR-80-109. 


METRIC  EXPLANATION  SUMMARY 


CRITERIA  . 

ACRONYM 

METRICS 

ACCURACY 

AY.l 

ACCURACY  CHECKLIST 

ANOMALY  MANAGEMENT 

AM.l 

ERROR  TOLERANCE/CONTROL  CHECK¬ 
LIST 

IMPROPER  INPUT  DATA  CHECKLIST 

IBiialB 

COMPUTATIONAL  FAILURES  CHECKLIST 

AM.4 

HARDWARE  FAULTS  CHECKLIST 

AM, 5 

DEVICE  ERRORS  CHECKLIST 

AM. 6 

COMMUNICATION  ERRORS  CHECKLIST 

AM.7 

NODE/COMMUNICATIONS  FAILURES 
CHECKLIST 

APPLICATION 

INDEPENDENCE 

AI.l 

DATA  BASE  SYSTEM  INDEPENDENCE 

DATA  STRUCTURE 

ARCHITECTURE  STANDARDIZATION 

AI.4 

MICROCODE  INDEPENDENCE 

AI.5 

ALGORITHM 

AUGMENTA8ILITY 

AG.  1 

DATA  STORAGE  EXPANSION  MEASURE 

■ 

COMPUTATION  EXTENSIBILITY  MEASURE 

CHANNEL  EXTENSIBILITY  MEASURE 

AG.  4 

DESIGN  EXTENSIBILITY  CHECKLIST 

AUTONOMY 

AU.l 

INTERFACE  COMPLEXITY  MEASURE 

AU.2 

SELF-SUFFICIENCY  CHECKLIST 

COMMONALITY 

CL.l 

COMMUNICATIONS  COMMONALITY 
CHECKLIST 

DATA  COMMONALITY  CHECKLIST 

E9i 

COMMON  VOCABULARY  CHECKLIST 

COMMUNICATIVENESS 

CM.  1 

USER  INPUT  INTERFACE  MEASURE 

CM.  2 

USER  OUTPUT  INTERFACE  MEASURE  ] 

_ l 

COMPLETENESS 

CP.l 

COMPLETENESS  CHECKLIST 

CONCISENESS 

CO.l 

HALSTEAD'S  MEASURE 

CONSISTENCY 

CS.l 

PROCEDURE  CONSISTENCY  MEASURE 

CS.2 

DATA  CONSISTENCY  MEASURE  ; 

DISTRIBUTEDNESS 

DL). 

DESIGN  STRUCTURE  CHECKLIST 

DOCUMENT  ACCESSIBILITY 

DA.  I 

ACCE55  NO-CONTROL 

WELL -STRUCTURED  DOCUMENTATION 

wmm 

SELECTIVE  USABILITY 

METRIC  EXPLANATION  SUMMARY 


CRITERIA 

ACRONYM 

METRICS 

EFFECTIVENESS 

EF.l 

PERFORMANCE  REQUIREMENTS 

EF.2 

ITERATIVE  PROCESSING  EFFICIENCY 
MEASURE 

EF.3 

DATA  USAGE  EFFICIENCY  MEASURE 

EF.4 

STORAGE  EFFICIENCY  MEASURE 

FUNCTIONAL  OVERLAP 

FO.l 

FUNCTIONAL  OVERLAP  MEASURE 

FUNCTIONAL  SCOPE 

FS.l 

FUNCTION  SPECIFICITY 

FS.2 

FUNCTION  COMMONALITY 

FS.3 

FUNCTION  COMPLETENESS 

GENERALITY 

GE.i 

MODULE  REFERENCE  BY  OTHER  MOD¬ 
ULES 

GE.2 

IMPLEMENTATION  FOR  GENERALITY 
CHECKLIST 

INDEPENDENCE 

ID.l 

SOFTWARE  SYSTEM  INDEPENDENCE  MEA¬ 
SURE 

ID. 2 

MACHINE  INDEPENDENCE  MEASURE 

MODULARITY 

■n 

MODULAR  IMPLEMENTATION  MEASURE 

MODULAR  DESIGN  MEASURE 

OPERABILITY 

OP  A 

OPERABILITY  CHECKLIST 

RECONFIGURABILITY 

RE.i 

RESTRUCTURE  CHECKLIST 

SELF-DESCRIPTIVENESS 

SD.l 

QUANTITY  OF  COMMENTS 

SD.2 

EFFECTIVENESS  OF  COMMENTS  MEASURE 

SD.3 

DESCRIPTIVENESS  OF  LANGUAGE  MEAS¬ 
URE 

SIMPLICITY 

SI.1 

DESIGN  STRUCTURE  MEASURE 

SI.2 

STRUCTURED  LANGUAGE  OR  PRE¬ 
PROCESSOR 

51.3 

DATA  AND  CONTROL  FLOW  COMPLEXITY 
MEASURE 

51.4 

CODING  SIMPLICITY  MEASURE 

SPECIFICITY 

5P.1 

SCOPE  OF  FUNCTION  MEASURE 

SYSTEM  ACCESSIBILITY 

SA.l 

ACCESS  CONTROL  CHECKLIST 

SA.2 

ACCESS  AUDIT  CHECKLIST 

C-3 


METRIC  EXPLANATION  SUMMARY 


CRITERIA 

ACRONYM 

METRICS 

SYSTEM  CLARITY 

5C.1 

INTERFACE  COMPLEXITY 

SC. 2 

PROGRAM  FLOW  COMPLEXITY 

SC.3 

APPLICATION  FUNCTIONAL  COMPLEXITY 

SC.4 

COMMUNICATION  COMPLEXITY 

5C.5 

STRUCTURE  CLARITY 

SYSTEM  COMPATIBILITY 

SY.l 

COMMUNICATION  COMPATIBILITY 
CHECKLIST 

SY.2 

DATA  COMPATIBILITY  CHECKLIST 

SY.3 

HARDWARE  COMPATIBILITY  CHECKLIST 

SY.4 

SOFTWARE  COMPATIBILITY  CHECKLIST 

SY.5 

DOCUMENTATION  FOR  OTHER  SYSTEM 

TRACEABILITY 

TR.I 

CROSS  REFERENCE 

TRAINING 

TN.l 

TRAINING  CHECKLIST 

VIRTUALITY 

VR.l 

SYSTEM/DATA  INDEPENDENCE  CHECK¬ 
LIST 

VISIBILITY 

VS.l 

MODULE  TESTING  MEASURE 

INTEGRATION  TESTING  MEASURE 

■D3HI 

SYSTEM  TESTING  MEASURE  | 

Criteria: 


Accuracy 


Metric: 


(1) 


(2) 


(3) 


(4) 


(5) 


(6) 


AY.l  Accuracy  Checklist. 

Each  element  is  a  binary  measure  indicating  existence  or  absence  of  the 
elements.  The  metric  is  the  sum  of  the  scores  of  the  following  applica¬ 
ble  elements  divided  by  the  number  of  applicable  elements. 

Error  analysis  performed  and  budgeted  to  module. 

An  error  analysis  must  be  part  of  the  requirements  analysis  performed  to 
develop  the  requirements  specification.  This  analysis  allocates  overall 
accuracy  requirements  to  the  individual  functions  to  be  performed  by  the 
system.  This  budgeting  of  accuracy  requirements  provides  definitive 
objectives  to  the  module  designers  and  implementers. 

A  definitive  statement  of  requirement  for  accuracy  of  inputs,  outputs, 
processing,  and  constants. 

See  explanation  (1)  above. 

Sufficiency  of  math  library. 

The  accuracy  of  the  math  library  routines  utilized  within  the  system  is 
to  be  checked  for  consistency  witn  the  overall  accuracy  objectives. 

Sufficiency  of  numerical  methods. 

The  numerical  methods  utilized  within  the  system  are  to  be  consistent 
with  the  accuracy  objectives. 

Execution  outputs  within  tolerances. 

A  final  measure  during  development  testing  is  execution  of  modules  and 
checking  for  accuracy  of  outputs. 

Accuracy  requirements  budgeted  to  functions/moduies. 

The  budgeting  of  accuracy  requirements  is  repeated  at  succeedingiy 
lower  levels  of  design  -  during  preliminary  and  detail  design. 


* 


C-5 


Criteria:  Anomaly  Management 


Metric:  AM. I  Error  Tolerance/Controi  Checklist. 

The  metric  is  the  sum  of  the  scores  given  to  the  following  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Concurrent  processing  centrally  controlled. 

Functions  which  may  be  used  concurrently  are  to  be  controlled  centrally 
to  provide  concurrency  checking,  read/write  locks,  etc.  Examples  are  a 
data  base  manager,  I/O  handling,  error  handling,  etc. 

(2)  Errors  fixable  and  processing  continued. 

When  an  error  is  detected,  the  capability  to  correct  it  on-line  and  then 
continue  processing  should  be  available.  An  example  is  an  operator 
message  that  the  wrong  tape  is  mounted  and  processing  will  continue 
when  correct  tape  is  mounted. 

(3)  When  an  error  condition  is  detected,  the  condition  is  to  be  passed  up  to 
calling  routine. 

The  decision  of  what  to  do  about  an  error  is  to  be  made  at  a  level 
where  an  affected  module  is  controlled.  This  concept  is  built  into  the 
design  and  then  implemented. 

(4)  Any  parallel  processing  centrally  controlled. 

When  parallel  processing  is  performed  it  is  controlled  by  concurrent 
inputs,  by  concurrent  output  checks,  and/or  by  comparing  output  results. 
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Criteria:  Anomaly  Management 

Metric:  AM.2  Improper  Input  Data  checklist. 

The  metric  is  the  sum  of  the  scores  given  to  the  following  elements 
divided  by  the  number  of  applicable  elements. 

(1)  A  definitive  statement  of  requirement  for  error  tolerance  of  input  data. 

Tlte  requirements  specification  must  identify  the  error  tolerance  cap¬ 
abilities  desired. 


(2)  Range  of  values  (reasonableness)  for  items  specified  and  checked. 

The  attribute  of  each  input  item  is  to  be  checked  for  reasonableness. 
Examples  are  checking  items  if  they  must  be  numeric,  alphabetic,  posi¬ 
tive  or  negative,  of  a  certain  length,  nonzero,  etc.  These  checks  are  to 
be  specified  at  design  and  exist  in  code  at  implementation. 

(3)  Conflicting  requests  and  illegal  combinations  identified  and  checked. 

Checks  to  see  if  redundant  input  data  agrees,  if  combinations  of  para¬ 
meters  are  reasonable,  and  if  requests  are  conflicting.  These  checks 
should  be  documented  in  the  design  and  exist  in  the  code  at  implementa¬ 
tion. 


(4)  All  input  is  checked  before  processing  begins. 

Input  checking  is  not  to  stop  at  the  first  error  encountered  but  to 
continue  through  all  the  input  and  then  report  all  errors.  Processing  is 
not  to  start  until  the  errors  are  reported  and  either  corrections  are 
made  or  a  continue  processing  command  is  given. 

(5)  Determination  that  all  data  is  available  prior  to  processing. 

To  avoid  going  through  several  processing  steps  before  incomplete  input 
data  is  discovered,  checks  for  sufficiency  of  input  data  are  to  be  made 
prior  to  the  start  of  processing. 
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Criteria: 


Anomaly  Management 


Metric: 


(1) 


(2) 


(3) 


(4) 


Metric: 


AM. 3  Computational  Failures  Checklist. 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

A  definitive  statement  of  requirement  for  recovery  from  computational 
failures. 

The  requirement  for  this  type  of  error  tolerance  capability  are  to  be 
stated  during  requirements  phase. 

Loop  and  multiple  transfer  index  parameters  range  tested  before  use. 
Range  tests  for  loop  indices  and  multiple  transfers  are  to  be  specified  at 
design  and  to  exist  in  code  at  implementation. 

Subscript  checking. 

Checks  for  legal  subscript  values  are  to  be  specified  at  design  and  coded 
during  implementation. 

Critical  output  parameters  reasonableness  checked  during  processing. 
Certain  range-of-value  checks  are  to  be  made  during  processing  to 
ensure  the  reasonableness  of  final  outputs.  This  is  usually  done  only  for 
critical  parameters.  These  are  to  be  identified  during  design  and  coded 
during  implementation. 

AM. 4  Hardware  Faults  Checklist. 

The  metric  is  the  sum  of  scores  from  the  applicable  elements  divided  by 
the  number  of  applicable  elements. 


Criteria*.  Anomaly  Management 


(1)  A  definitive  statement  of  requirements  for  recovery  from  hardware 
faults. 

The  handling  of  hardware  faults  such  as  arithmetic  faults,  power  failure, 
clock  interrupt,  etc.,  are  to  be  specified  during  the  requirements  phase. 

(2)  Recovery  from  hardware  faults. 

The  design  specification  and  code  to  provide  the  recovery  from  the 
hardware  faults  identified  in  the  requirements  must  exist  in  the  design 
and  implementation  phases  respectively. 

Metrics  AM.5  Device  Errors  Checklist. 

The  metric  is  the  sum  of  the  scores  given  to  the  following  applicable 
elements  divided  by  the  number  of  applicable  elements. 

(1)  A  definitive  statement  of  requirements  for  recovery  from  device  errors. 

The  handling  of  device  errors  such  as  unexpected  end-of-files  or  end-of- 
tape  conditions  and  read/write  failures  are  specified  during  the  require¬ 
ments  phase. 

(2)  Recovery  from  device  errors. 

The  design  specification  and  code  to  prov  de  the  required  handling  of 
device  errors  must  exist  in  the  design  and  implementation  phases  respec¬ 
tively. 

Metrics  AM. 6  Communications  Errors  Checklist. 

The  metric  is  the  sum  of  the  scores  given  to  the  following  elements 
divided  by  the  number  of  applicable  elements. 

(1)  A  definitive  statement  of  requirements  for  recovery  from  communication 
errors. 

Explicit  requirements  are  to  be  stated  for  recovery  from  communication 
errors. 
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Criteria!  Anomaly  Management 


(2)  Provisions  for  recovery  from  communication  errors. 

The  preliminary  design  should  reflect  a  design  solution  to  the  stated 
requirements. 

(3)  Check  sums  computed  and  transmitted  with  all  messages. 

Check  sums  are  a  common  form  of  detecting  communication  errors. 

(4)  Check  sums  computed  and  compared  upon  message  reception. 

Check  sums  are  a  common  form  of  detecting  communication  errors. 

Metric:  AM. 7  Node/Communications  Failures  Checklist. 

The  metric  is  the  sum  of  the  scores  given  to  the  following  elements 
divided  by  the  number  of  applicable  elements. 


(1)  A  definitive  statement  of 

node/communication  failures. 

requirements 

for 

recovery 

from 

Explicit  requirements  are  to 

node/communication  failures. 

be  stated 

for 

recovery 

from 

(2)  Provisions  for  recovery  from  node/communication  failures. 

The  preliminary  design  should  reflect  a  design  solution  to  the  stated 
requirements. 

(3)  Adjacent  nodes  checked  for  operational  status. 

C(.’  king  adjacent  nodes  is  a  common  form  of  detecting  node  failures. 

(4)  Alternate  strategies  for  message  routing. 

Employing  an  alternate  message  routing  strategy  is  a  common  way  of 
recovering  from  node/communication  failures. 
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Criteria* 


Application  Independence 


Metric: 

(1) 

Metric: 

(1) 

(2) 

(3) 


Al.l  Database  System  Independence. 

Software  which  is  free  from  database  system  reference  has  higher  reus¬ 
ability. 

The  metric  measure  is  based  on  how  the  module  is  Independent  of  the 
database  system. 

Free  from  database  system  reference. 

The  metric  is  based  on  the  database  system  reference  within  a  module. 

AI.2  Data  Structure. 

Generalized  data  structures  which  are  easy  to  understand,  flexible,  and 
extensible  reduce  the  costs  associated  with  reusing  the  software.  The 
software  with  control  of  data  structure  has  enhanced  modifiability,  and 
it  tends  to  be  more  reusable.  The  metric  is  the  sum  of  the  scores  of 
the  following  applicable  elements  divided  by  the  number  of  applicable 
elements. 

Data  in  parameter  list,  data  structure  described  parametrically. 

Parametric  definitions  of  data  structures  will  reduce  the  reuse  software 
costs.  The  metric  is  based  on  how  many  data  items  could  be  para¬ 
metrized  and  parametrized  data  items. 

Data  communicated  through  common  storage  region  and  with  adequate 
comments. 

To  reduco  the  software  reuse  costs  the.  data  should  be  centrally  con¬ 
trolled  such  as  through  global  storage.  Then  common  data  in  a  module 
must  have  adequate  explanations.  This  is  a  binary  measure. 

Control  of  database  structures,  both  global  and  local,  i.e.,  all  data 
centrally  controlled  and  symbolically  defined  and  referenced. 

See  explanation  for  (2)  above. 
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Criteria* 


Application  Independence 


(*> 


(5) 


Metric* 


(l) 


(2) 


(3) 


Logical  processing  independent  of  data  storage  specification  and  require* 
ment. 

The  software  with  logical  processing  independent  of  data  storage  will 
tend  to  be  more  reusable.  The  measure  is  based  on  the  number  of 
modules  which  do  not  comply. 

Each  module  has  code  comments  about  data  items  description  including 
global  4  parameter  input/output  and  local  variables. 

See  explanation  for  (2)  above. 

AI.3  Architecture  Standardization. 

Standardization  of  computer  architecture  can  increase  the  potential 
reuse  of  software  by  increasing  the  number  of  environments  in  which  the 
software  can  be  executed  without  change.  The  metric  is  the  sum  of  the 
scores  of  the  following  applicable  elements  divided  by  the  number  of 
applicable  elements. 

Module  is  free  from  computer  architecture  reference. 

When  software  is  independent  from  computer  architecture  reference  it 
tends  to  be  more  reusable.  This  is  a  binary  measure. 

Module  is  in  standard  M  bits  computer  architecture  (Nebula). 

When  software  is  in  a  standard  computer  architecture  then  it  will  be 
easier  to  reuse  in  another  computer  with  standard  architecture.  This  is 
a  binary  measure. 

Code  statements  are  free  from  machine  architecture. 

See  explanation  for  (1)  above. 
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Criteria:  Application  Independence 

Metric:  AI.4  Microcode  Independence. 

Using  the  microcode  or  machine  language  code  in  software  will  reduce 
the  number  of  environments  where  software  can  be  reused  and  also 
reduce  the  software  flexibility.  The  metric  measure  is  based  on  how  the 
module  is  free  from  microcode  instructions. 

(1)  Number  of  modules  used  microcode  instruction. 

The  metric  is  based  on  the  microcode  references  within  a  module. 

Metric:  AI.5  Algorithm. 

An  algorithm  that  functions  well  over  a  wide  range  of  inputs  will 
generally  require  less  modification  before  it  can  be  reused.  The  use  of 
table  driven  algorithms  will  produce  highly  reusable  softv»are  which  can 
be  easily  adapted  to  different  applications.  The  metric  is  the  sum  of 
the  scores  of  the  following  applicable  elements  divided  by  the  total 
number  of  applicable  elements. 

(1)  Valid  range. 

The  range  of  inputs  the  function  algorithm  can  handle.  The  metric  .s 
based  on  the  number  of  the  domains  the  algorithm  works  for. 

(2)  Is  the  algorithm  table  driven? 

The  table-driven  algorithm  can  be  easily  adapted  to  different  applica¬ 
tions.  'rhe  metric  is  a  binary  measure. 

(3)  Is  the  algorithm  certification  available? 

The  software  with  algorithm  certification  available  tends  to  be  more 
reusable.  The  metric  is  a  binary  measure. 

(4)  Is  the  algorithm  test  data  available? 

See  explanation  for  (3)  above. 
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Criteria!  Application  Independence 


(5)  Each  module  has  code  comments  about  algorithm  description. 

The  algorithm  usage  should  be  explained  in  the  code  comments.  The 
measure  is  based  on  the  number  of  modules  which  do  not  follow  this 
practice. 


Criteria: 


Augmentability 


Metric: 

(1) 

(2) 

(3) 

Metric: 

(1) 


AG.l  Data  Storage  Expansion  Measure. 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

Logical  processing  independent  of  storage  specification/requirements. 
The  logical  processing  of  a  module  is  to  be  independent  of  storage  size, 
buffer  space,  or  array  size*.  The  design  provides  for  variable  dimensions 
and  dynamic  array  sizes  to  be  defined  parametrically.  The  metric  is 
based  on  the  number  of  modules  containing  hard-coded  dimensions  which 
do  not  exemplify  this  concept. 

Percent  of  memory  capacity  uncommitted. 

The  amount  of  memory  available  for  expansion  is  an  important  measure. 
This  measure  identifies  the  percent  of  available  memory  which  has  not 
been  utilized  in  implementing  the  current  system. 

Percent  auxilliary  storage  capacity  uncommitted. 

See  explanation  for  (2)  above. 

AG. 2  Computation  Extensibility  Measure. 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

Accuracy,  convergence,  timing  attributes  which  control  processing  are 
parametric. 

A  module  which  can  provide  varying  degrees  of  convergence  or  timing  to 
achieve  greater  precision  provides  this  attribute  of  extensibility.  Hard¬ 
coded  control  parameters,  counters,  clock  values,  etc.  violate  this  meas¬ 
ure.  This  measure  is  based  on  the  number  of  modules  which  do  not 
exemplify  this  characteristic. 


Criteria: 


Augmcntability 


(2)  Modules  table  driven. 

The  use  of  tables  within  a  module  facilitates  different  representations 
and  processing  characteristics.  This  measure  which  can  be  applied 
during  design  and  implementation  h  based  on  the  number  of  modules 
which  are  not  table  driven. 

(3)  Percent  of  speed  capacity  uncommitted. 

A  certain  function  may  be  required  in  the  performance  requirements 
specification  to  be  accomplished  in  a  specified  time  for  overall  timing 
objectives.  The  amount  of  time  not  used  by  the  current  implementation 
of  the  function  is  procescing  time  available  for  potential  expansion  of 
computational  capabilities.  This  measure  identifies  the  percent  of  total 
processing  time  that  is  uncommitted. 

Metric:  AG. 3  Channel  Extensibility  Measure. 

The  metric  is  the  sum  of  the  scores  given  to  the  following  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Spare  I/O  channel  capacity  (by  peripheral). 

A  load  will  be  placed  on  the  channels  to  each  peripheral  because  of  the 
design  solution.  The  amount  of  channel  capacity  which  is  uncommitted 
is  the  amount  available  for  potential  expansion. 

(2)  Spare  communication  channel  capacity. 

A  load  will  be  placed  on  each  communication  channel  because  of  the 
design  solution.  The  amount  of  communication  channel  capacity  which  is 
uncommitted  is  the  amount  available  for  potential  expansion. 

Metric:  AG.4  Design  Extensibility  Checklist. 

The  metric  is  the  sum  of  the  scores  given  to  the  following  elements 
divided  by  the  number  of  applicable  elements. 
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Criteria: 


Auamentabllity 


(1)  Processors,  communication  links,  memory  devices,  and  peripherals  com¬ 
patible  (o f  a  common  vendor  or  model). 

It  is  desirable  to  have  network  hardware  compatible  as  this  minimizes 
interface  complexity  and  eases  the  task  of  expansion. 

(2)  Documentation  reveals  performance  price  of  software/system  for  en¬ 
hancement  trades. 

The  cost  required  to  achieve  the  specified  performance  levels  has  seldom 
been  documented;  yet  this  is  an  essential  element  in  performing  trades 
for  enhancing  the  system. 

(3)  Specifications  identify  new  technology  tradeoff  areas  for  software. 

This  information  would  be  useful  for  future  changes  in  the  software  and 
the  system. 

(4)  Software  specifications  include  requirements  for  the  criteria  of  the  qual¬ 
ity  factor  expandability. 

Building  in  the  expansion  capability  will  minimize  future  costs. 
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Criteria: 


Autonomy 


Metric: 


(i) 


(2) 


(3) 


(4) 


Metric: 


(1) 


AU.  1  Interface  Complexity  Measure. 

The  metric  is  the  sum  of  the  scores  given  to  the  following  elements 
divided  by  the  number  of  applicable  elements. 

Processes/functions  separated  as  logical  "wholes"  to  minimize  interface 
complexity. 

Minimizing  interface  complexity  in  the  functional  design  will  aid  in 
keeping  interfaces  simple  in  the  detail  design. 

Interface  code. 

The  greater  the  amount  of  interface  code,  in  general,  the  more  complex 
is  the  interface.  This  measure  identifies  the  fraction  of  non-interface 
code. 

Interface  modules. 

The  greater  number  of  interface  modules,  in  general,  the  more  complex 
is  the  interface.  This  measure  identifies  the  fraction  of  non-interface 
modules. 

Communication  loading. 

The  complexity  of  the  interface  is  reflected  in  part  by  the  percentage  of 
use. 

This  measure  identifies  the  fraction  of  idle  interface  communication 
time. 

AU.2  Self-sufficiency  Checklist. 

The  metric  is  the  sum  of  the  scores  given  to  the  following  elements 
divided  by  the  number  of  applicable  elements. 

Software  volatility  -  each  CPU/system  has  separate  power  supply. 

System  software  vulnerability  is  reduced  by  increasing  the  independence 
of  each  CPU/system. 
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Criteriat  Autonomy 


(2)  Each  scheduling  unit  (i.e.,  executive,  operating  system)  tests  its  own 
operation,  communication  links,  memories,  and  peripherals. 

System  software  vulnerability  i«  reduced  through  independent  node  self¬ 
test. 

(3)  Software  system  includes  word-processing  capability. 

System  autonomy  is  enhanced  by  being  able  to  produce  documentation 
on-site. 


Metrict  CL.l  Communications  Commonality  Checklist. 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 


(1)  Definitive  statement  of  requirements  for  communication  with  other  sys¬ 
tems. 

During  the  requirement  phase,  the  communication  requirements  with 
other  systems  must  be  considered.  This  is  a  binary  measure  of  the 
existence  of  this  consideration. 


(2)  Protocol  standards  established  and  followed  for  network  process  control. 
The  communication  protocol  standards  for  communication  with  other 
system?  are  to  be  established  during  the  design  phase  and  followed 
during  implementation.  This  binary  measure  applied  at  each  of  these 
phases  indicates  whether  the  standards  were  established  and  followed. 

(3)  Single  module  interface  for  input  (from  another  system). 

The  more  modules  which  handle  input  the  more  difficult  it  is  to  inter¬ 
face  with  another  system  and  implement  standard  protocols.  This  meas¬ 
ure  is  based  on  the  reciprocal  of  the  number  of  modules  which  handle 
input. 

(4)  Single  module  interface  for  output  (to  another  system). 

For  simikr  reasons  as  (3)  above  this  measure  is  the  reciprocal  of  the 
number  of  output  modules. 

(5)  Specific  requirements  for  network  process  control. 

Network  process  control  requirements  should  be  specified  during  the 
requirements  analysis  phase  and  consider  all  nodes  in  the  network. 


Criteria* 


(g)  Specific  requirements  for  user  section  control. 

Requirements  for  the  control  cf  a  user  session  on  the  network  should  be 
specified  during  the  requirements  analysis  phase  and  consider  all  nodes  in 
the  network. 

(7)  Specific  requirements  tor  communication  routing  strategy. 

Requirements  for  communication  routing  should  be  specified  during  the 
requirements  analysis  phase  and  consider  ail  nodes  in  the  network  con- 
figuration. 

(8)  Protocol  standards  established  and  followed  for  user  session  control. 

The  design  and  implementation  should  comply  with  network-wide  proto¬ 
col  standards. 

(9)  Protocol  standards  established  and  followed  for  communication  routing. 

The  design  and  implementation  should  comply  with  network-wide  proto¬ 
col  standards. 

(10)  Number  of  systems  responding  correctly  to  successfully  complete  hand¬ 
shaking.  The  larger  the  number  of  systems  which  must  respond  correct¬ 
ly,  the  greater  the  effort  required. 

(11)  Low  time  dependency  on  handshaking.  High  time  dependencies  impose 
greater  constraints  on  computation  and  response  times,  which  will  in¬ 
crease  the  total  effort, 

(12)  No  communication  time  dependency. 

If  the  communication  function  has  time  dependencies,  such  as  freshness 
of  data  or  response  to  input  data  within  certain  time  limits,  then  the 
effort  increases. 
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Criteria: 

(13) 


Number  of  other  systems  this  system  wiil  interface  with. 

The  number  of  systems  with  which  this  system  must  interoperate  should 
greatly  affect  the  total  interoperability  effort. 


(14)  No  timing  dependency  on  data  freshness. 

The  requirement  for  data  freshness  will  increase  effort  to  meet  timing 
factors. 


(13)  Operating  procedures  known. 

The  operating  procedures  used  with  the  system  must  be  known  so  the 
requirements  can  be  understood  in  context. 

Metric:  CL.2  Data  Commonality  Checklist. 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Definitive  statement  for  standard  data  representation  for  communication 
with  other  systems. 

This  is  a  binary  measure  of  the  existence  of  consideration  for  standard 
data  representation  between  systems  which  are  to  be  interfaced.  This 
must  be  addressed  and  measured  in  the  requirements  phase. 

(2)  Translation  standards  among  representations  established  and  followed. 

More  than  one  translation  from  the  standard  data  representations  used 
for  interfacing  with  other  systems  may  exist  within  a  system.  Standards 
for  these  translations  are  to  be  established  and  followed.  This  binary 
measure  identifies  if  the  standards  are  established  during  design  and 
followed  during  implementation. 

(3)  Single  module  to  perform  each  translation. 

This  measure  is  the  reciprocal  of  the  maximum  number  of  modules  which 
perform  a  translation. 
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Criteria:  Commonality 


Metric:  CL.3  Common  Vocabulary  Checklist. 

The  binary  metric  is  the  single  value  answer  to  the  question  of  common 
vocabulary  use  among  Interoperating  systems.  If  there  Is  more  than  one 
system  with  which  the  subject  system  is  to  interoperate,  then  the  value 
of  this  metric  is  the  average  of  the  individual  metrics  for  each  inter¬ 
operating  system. 

(1)  Do  both  projects  use  the  same  technical  vocabulary  with  identical  mean¬ 
ings?  According  to  published  material  on  interoperability,  one  of  the 
most  prevalent  and  pervasive  problems  is  the  use  of  inconsistent  termin¬ 
ologies.  Projects  may  use  different  vocabularies  with  the  same  mean¬ 
ings,  or  use  the  same  vocabulary  with  different  meanings*  As  a  result, 
people  either  don't  understand  each  other  and  know  it,  or  don't  under¬ 
stand  each  other  and  don't  know  it.  Either  way,  interoperability  pro¬ 
blems  are  the  sum  result. 
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Criteria* 


Communicativeness 


Metrict  CM. I  User  input  Interlace  Measure. 

The  metric  is  the  sum  ot  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Default  values  defined. 

A  method  of  minimizing  the  amount  of  input  tequired  is  to  provide 
defaults.  This  measure,  applied  during  design  and  implementation,  is 
based  on  the  number  of  defaults  allowed  divided  by  the  total  number  of 
input  parameters. 

(2)  Input  format  uniform. 

The  greater  the  number  of  input  formats  there  are  the  more  difficult 
the  system  is  to  use.  This  measure  is  based  on  the  total  number  of  input 
formats. 


(3)  Each  input  record  self-identif  ying. 

Input  records  which  have  self 'identifying  codes  enhance  the  accuracy  of 
user  inputs.  This  measure  is  based  on  the  number  of  input  records  that 
are  not  self  identifying  divided  by  die  total  number  of  input  records. 

(4)  Input  can  be  verified  by  user  prior  to  execution. 

The  capability,  displaying  input  upon  request  or  echoing  the  input  auto¬ 
matically,  enables  the  use.'  to  check  his  inputs  before  processing.  This  is 
a  binary  measure  of  the  existence  of  the  design  and  implementation  of 
this  capability. 

(5)  Input  terminated  by  explicitly  defined  logical  end  of  input. 

The  user  should  not  have  to  provide  a  count  of  input  cards.  This  is  a 
binary  measure  of  the  design  and  implementation  of  this  capability. 
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Criteria*  Communtcativenew 


(6)  Provision  for  specifying  input  from  different  media. 

The  flexibility  of  input  must  be  decided  during  the  requirements  analysis 
phase  and  followed  through  during  design  and  implementation.  This  is  a 
binary  measure  of  the  existence  of  the  consideration  of  this  capability 
during  all  three  of  these  phases. 

Metrics  CM.2  User  Output  Interface  Measure. 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Selective  output  controls. 

The  existence  of  a  requirement  lor,  design  for,  and  implementation  of 
selective  output  controls  is  indicated  by  this  binary  measure.  Selective 
controls  include  choosing  specific  outputs,  output  formats,  amount  of 
output,  etc. 

(2)  Outputs  have  unique  descriptive  user  oriented  labels. 

This  is  a  binary  measure  of  the  design  and  implementation  of  unique 
output  labels.  In  addition,  the  labels  are  *o  be  descriptive  to  the  user. 
This  includes  not  only  the  labels  which  are  used  to  reference  an  output 
report  but  also  the  title,  column  headings,  etc.  within  that  report. 

(3)  Outputs  have  user  oriented  units. 

This  is  a  binary  measure  which  extends  (2)  above  to  the  individual  output 
items. 

(4)  Uniform  output  labels. 

This  measure  corresponds  to  (2)  above  and  is  the  reciprocal  of  the 
number  of  different  output  formats. 


Criteria: 


Communicativeness 


(5)  Logical  groups  of  output  separated  for  user  examination. 

Utilization  of  top  of  page,  blank  lines,  lines  of  asterisks,  etc.,  provide 
for  easy  identification  of  logically  grouped  output.  This  binary  measure 
identifies  if  these  techniques  are  used  during  design  and  implementation. 

(6)  Relationship  between  error  messages  and  outputs  is  unambiguous. 

This  is  a  binary  measure  applied  during  design  and  implementation  which 
identifies  if  error  messages  will  be  directly  related  to  the  output. 

(7)  Provision  for  redirecting  output  to  different  media. 

This  is  a  binary  metric  which  identifies  if  consideration  is  given  to  the 
capability  to  redirect  output  to  different  media  during  requirements 
analysis,  design,  and  implementation. 

(8)  Standard  user  interfaces  for  network  information  and  data  across. 

This  is  a  binary  metric  which  considers  a  common  user  language  for 
accessing  information/data  throughout  the  network.  This  capability  re¬ 
lieves  the  user  of  the  need  to  know  the  languages  of  different  nodes. 
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Criteria: 


Complete  ness 


Metric* 


(1) 


(2) 


(3) 


(4) 


(5) 


CP.l  Completeness  Checklist. 

This  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

Unambiguous  requirements/references  for  input,  function,  and  output. 
Unique  references  to  data  or  functions  avoid  ambiguities  such  as  a 
function  being  called  one  name  by  one  module  and  by  another  name  by 
another  module.  Unique  references  avoid  this  type  of  ambiguity  in  all 
three  phases. 

All  data  references  defined,  computed,  or  obtained  from  an  external 
source. 

Each  data  element  is  to  have  a  specific  origin.  At  the  requirements 
level  only  major  global  data  elements  and  a  few  specific  local  data 
elements  may  be  available  to  be  checked.  The  set  of  data  elements 
available  for  completeness  checking  at  the  design  level  increases  sub¬ 
stantially  and  is  to  be  complete  at  implementation. 

All  defined  functions  used. 

A  function  which  is  defined  but  not  used  during  a  phase  is  either 
nonfunctional  or  a  reference  to  it  has  been  omitted. 

All  referenced  functions  defined. 

A  system  is  not  complete  at  any  phase  if  dummy  functions  are  present 
or  if  functions  have  been  referenced  but  not  defined. 

AU  conditions  and  processing  defined  for  each  decision  point. 

Each  decision  point  is  to  have  all  of  its  conditions  and  alternative 
processing  paths  defined  at  each  phase  of  the  software  development. 
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Criteria:  Completeness 


The  level  of  detail  to  which  the  conditions  and  alternative  processing  are 
described  may  vary  but  the  important  element  is  ,  tat  all  alternatives 
are  described. 

(6)  All  defined  and  referenced  calling  sequence  parameters  agree. 

For  each  interaction  between  modules,  the  full  complement  of  defined 
parameters  for  the  interface  is  to  be  ur  :d.  A  particular  call  to  a 
module  should  not  pass,  for  example,  only  five  of  the  six  defined  para¬ 
meters  for  that  module. 

(7)  All  problem  reports  resolved. 

At  each  phase  in  the  development,  problem  reports  are  generated.  Each 
is  to  be  closed  or  a  reolution  indicated  to  ensure  a  complete  product. 
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Criteria:  Conciseness 


Metric:  CO.l  Halstead's  Measure. 

The  metric  is  based  on  Halstead's  concept  of  length  (HALSM77). 

The  observed  length  of  a  module  is 
Nq  =  N|  +  N2  where: 

=  total  usage  of  all  operands  in  a  module 
N2  =  total  usage  of  all  operands  in  a  module 

The  calculated  length  of  a  module  is 

Nc  =  nilog2nj  +  n2log2n2  where: 

ni  =  number  of  unique  operators  in  a  module 

n2  =  number  of  unique  operators  in  a  module 

The  metric  is  normalized  as  follows: 

Nc-Nq 

1  -  Nq  or, 

Nc  ~  Np 

0  if  n0  greater  than  1 

At  a  system  level  the  metric  is  the  averaged  value  of  all  the  module  metric  values. 


Criteria: 


Consistency 


Metric:  CS.i  Procedure  Consistency  Measure. 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Standard  design  representation. 

Flow  charts,  HIPO  charts,  Program  Design  Language  -  whichever  form  of 
design  representation  is  used,  standards  for  representing  the  elements  of 
control  flow  are  to  be  established  and  followed.  This  element  applies  to 
design  oniy.  The  measure  is  based  on  the  number  of  modules  whose 
design  representation  does  not  comply  with  the  standards. 

(2)  Calling  sequence  conventions. 

Interactions  between  modules  are  to  be  standardized.  The  standards  ate 
to  be  established  during  design  and  followed  during  implementation.  The 
measure  is  based  on  the  number  of  modules  which  do  not  comply  with 
the  conventions. 


(3)  Input/output  conventions. 

Conventions  for  which  modules  will  perform  I/O,  how  it  will  be  accom¬ 
plished,  and  the  I/O  formats  are  to  be  established  and  followed.  The 
measure  is  based  on  which  modules  do  not  comply  with  the  conventions. 

(4)  Error  handling  conventions. 

A  consistent  method  for  error  handling  is  required.  Conventions  estab¬ 
lished  in  design  are  followed  into  implementation.  The  measure  is  based 
on  the  number  of  modules  which  do  not  comply  with  the  conventions. 
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Criteria!  Consistenc 


Metric:  CS.2  Data  Consistency  Measure. 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Standard  data  usage  representation. 

In  concert  with  CS.l  (1),  a  standard  design  representation  for  data  usage 
is  to  be  established  and  followed.  This  is  a  design  metric  only,  iden¬ 
tifying  the  number  of  modules  which  violate  the  standards. 

(2)  Naming  conventions. 

Naming  conventions  for  variables  and  modules  are  to  be  established  and 
followed. 

(3)  Consistent  global  definitions. 

Global  data  elements  are  to  be  defined  in  the  same  manner  by  all 
modules.  The  measure  is  based  on  the  number  of  modules  in  which  the 
global  data  elements  are  defined  in  an  inconsistent  manner  for  both 
design  and  implementation. 

(4)  Requirements  for  verifying  database  consistency/concurrency. 

In  a  system  where  multiple  versions  of  the  same  information  and  data 
exist  at  different  nodes,  requirements  should  be  stated  to  verify  consis¬ 
tency  and  concurrency  of  the  multiple  versions. 

(5)  Procedures  for  verifying  database  consistency/concurrency. 

As  in  (4)  above,  procedures  should  be  developed  for  verifying 
consistency/concurrency  of  multiple  versions. 


Criteria: 


Qistributcdness 


Metric: 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 


DI.l  Design  Structure  Checklist. 

The  metric  is  the  sum  of  the  scores  given  to  the  following  elements 
divided  by  the  number  of  applicable  elements. 

Design  organization  identifies  ail  functions  and  interfaces. 

Identification  of  the  complete  set  of  functions  and  interfaces  is  essential 
to  the  design. 

Database  organization  identifies  all  data  and  data  flow. 

Identification  of  the  complete  set  of  data  and  flows  is  essential  to  the 
design. 

Specific  requirements  for  information  distribution  within  the  database. 
Early  decisions  are  required  on  how  to  distribute  information  within  a 
network. 

Provisions  for  file/library  access  from  other  nodes. 

Network  nodes  will  rely  on  other  nodes  for  some  information  or  for 
backup  data. 

Provisions  for  selecting  alternate  processing  capabilities. 

A  versatile  network  design  will  provide  alternate  processing  sources. 

Critical  system  functions  distributed  over  redundant  elements/nodes. 

System  vulnerabilty  is  reduced  by  distributing  critical  functions  across 
different  nodes. 

Distribution  of  control  functions  ensures  network  operation/integrity 
under  anomalous  conditions. 

Again,  a  good  network  design  will  take  advantage  of  the  redundant 
processing  capability  and  distribute  network  control  functions  across 
different  nodes. 
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Criteria:  Distributedness 


(8)  Logical  structure  and  function  separated  in  the  design. 

Logical  entities  can  be  grouped  under  one  function  or  can  be  separated 
among  several  functions.  It  is  important  to  distinguish  between  logical 
structure  and  function. 

(9)  Physical  structure  and  function  separated  in  the  design. 

Functions  can  be  grouped  within  one  physical  structure  or  can  be  separa¬ 
ted  among  several  physical  structures.  It  is  important  to  distinguish 
between  physical  structure  and  function. 

(10)  Number  of  nodes  that  can  be  removed  and  still  have  each  node  able  to 
communicate  with  each  remaining  node  (Kleitman's  algorithm). 

The  node  connectivity  is  the  minimum  number  of  nodes  whose  removal 
will  disconnect  the  two  nodes.  If  the  two  nodes  have  an  arc  linking 
them,  there  is  no  way  to  disconnect  them  by  removing  nodes,  not  even 
by  removing  all  n  —  2  of  the  remaining  nodes  in  an  n  node  network.  In 
this  case  the  node  connectivity  is  defined  as  n  —  1.  If  a  network  can 
withstand  the  loss  of  k  nodes,  it  can  also  withstand  the  loss  of  k  links, 
by  Whitney's  theorem.  An  algorithm  due  to  Xleitman  (1969)  is  as 
follows.  Pick  any  node  at  random  and  call  it  and  every  other  node  in 
the  network  is  at  least  k  +  1. 

Now  delete  Nj  and  aii  its  attached  links  from  the  network  and  choose 
another  node,  Nj-  Verify  that  this  node  has  at  least  a  node  connectivity 
of  k  with  every  other  node.  Next,  remove  N^  and  its  attached  links 
from  the  network  and  choose  a  third  node,  N y  Verify  that  N^  has  at 
least  a  node  connectivity  of  k  —  I  with  each  of  the  remaining  nodes. 
Continue  this  process  until  you  have  verified  that  some  node  N^  +  j  is 
1 -connected  to  all  nodes  of  the  remaining  network.  At  this  point  the 
algorithm  terminates. 

Kleitman,  D.:  "Methods  for  Investigating  the  Connectivity  of  Large 
Graphs,"  IEEE  Trans.  Circuit  Theory,  vol.  CT  16,  pp.  232-233,  May 
1969. 


Criteria:  Distributedness 


S.  Even  (1973)  has  devised  another  way  to  check  for  connectivity  k. 

Even,  5.:  Graph  Algorithms.  Potomac,  Md.:  Computer  Science  Press, 
1979. 

Even,  S.:  "An  Algorithm  lor  Determining  Whether  the  Connectivity  of  a 
Graph  Is  at  Least  k,"  SIAM  3.  Comput.,  vol.  4,  pp.  393-396,  Sept. 
1975. 


Criteria: 


Document  Accessibility 


Metric: 

(1) 

Metric: 

(1) 

(2) 

(3) 

(4) 

(5) 


DA.  I  Access  No-Control. 

Is  there  no  access  control  to  the  software  document? 

This  metric  provides  a  measure  of  the  ease  of  access  to  software 
documents. 

DA. 2  Well-Structured  Documentation. 

The  metric  is  the  sum  of  the  following  applicable  elements  divided  by 
the  number  of  applicable  elements. 

Clearly  and  simply  written  documents. 

When  the  documents  are  the  more  clearly  and  simply  written,  the  soft¬ 
ware  programs  are  the  easier  to  understand  and  are  more  useful.  This  is 
a  binary  measure. 

Neat  and  carefully  drawn  software  flow  charts  with  adequate  informa¬ 
tion  and  explanation. 

When  the  documents  provide  system  software  flow  charts  and  explain  the 
functions  performed,  they  are  more  useful.  This  is  a  binary  measure. 

Hierarchical  structured  table  of  contents  used  in  documents. 

The  documents  with  hierarchical  structure  will  make  it  easy  to  skim 
through  until  the  desired  information  is  found,  then  read  in  detail.  Then 
the  information  in  the  documents  is  more  accessible.  This  is  a  binary 
measure. 

Inde;-  system  used  in  documents. 

Documents  with  an  index  system  will  make  it  easier  and  faster  to  locate 
the  required  information.  Then  the  contents  of  the  documents  are  more 
accessible.  This  is  a  binary  measure. 

Separate  volumes  based  on  function  provided. 

See  explanation  for  (3)  and  (4)  above. 
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Criteria: 


Document  Accessibility 


(6)  Provide  global  information  about  the  functional  range  of  the  svstem. 

The  documents  should  have  global  information  about  the  range  of  the 
function  performed.  Then  the  documents  are  more  useful.  This  is  a 
binary  measure. 

(7)  Describe  the  functions  performed. 

The  documents  should  describe  the  functions  performed  in  the  system. 
This  is  a  binary  measure. 

(8)  Describe  the  algorithm  used  and  limitations. 

The  documents  should  describe  the  algorithm  and  their  limitations.  Then 
the  user  will  know  if  they  are  applicable  or  not  for  the  desired  applica¬ 
tion.  This  is  a  binar  y  measure. 

(9)  Describe  the  relationship  between  functions. 

The  documents  should  describe  the  relationship  between  the  functions. 
Then  the  documents  will  be  more  useful.  This  is  a  binary  measure. 

(10)  Provide  software  program  listing. 

The  documents  should  contain  the  program  source  listing.  Then  the 
information  in  the  documents  is  complete.  This  is  a  binary  measure. 

Metric:  DA. 3  Selective  Usability 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Options  available  to  the  user  so  that  selected  computation  or  output 
feature  may  be  requested. 

The  software  with  these  options  tends  to  be  more  reusable.  This  is  a 
binary  measure. 
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Criteria!  Document  Accessibility 

(2)  Modules  allow  lor  modifying  resource  utilization  i.e.,  through  use  vari¬ 
able  dimensioned  arrays. 

The  software  allowing  resource  utilization  modification  tends  to  be  more 
reusable.  This  is  a  binary  measure. 

(3)  Required  new  functions  can  be  satisfied  by  using  existing  design. 

The  required  functions  for  the  new  application  can  generally  be  satisfied 
by  adaptation  of  functions/modules  from  the  existing  design.  The  meas¬ 
ure  is  based  on  the  number  of  existing  functions  associated  with  the 
required  new  functions.  This  is  an  application-dependent  metric. 


Criteria: 


Effectiveness 


Metric: 


Metric: 


(1) 


(2) 


(3) 


(4) 


EF.  1  Performance  Requirements. 

Performance  requirements  and  limitations  specified  and  allocated  to 
functions/design. 

Performance  requirements  for  the  system  must  be  broken  down  and  allo¬ 
cated  appropriately  to  the  functions  and  mooules  during  the  design.  This 
metric  simply  identifies  if  the  performance  requirements  have  (l)  or 
have  not  (0)  been  allocated  during  the  design. 

EF.2  Iterative  Processing  Efficiency  Measure. 

The  metric  at  the  module  level  is  the  sum  of  the  scores  of  the  following 
applicable  elements  divided  by  the  number  of  elements.  At  the  system 
level  it  is  an  averaged  score  for  ail  of  the  modules. 

Non-loop  dependent  computations  kept  out  of  loop. 

Such  practices  as  evaluating  constants  in  a  loop  are  to  be  avoided.  This 
measure  is  based  on  the  number  of  non-loop  dependent  statements  found 
in  all  loops  in  a  module.  This  is  to  be  measured  from  a  detailed  design 
representation  during  design  and  from  the  code  during  implementation. 

Performance  optimizing  compiier/assembly  language  used. 

This  is  a  binary  measure  which  identifies  if  a  performance  optimizing 
compiler  was  used  (l);  or  if  assembly  language  was  used  to  accomplish 
performance  optimization  (1);  or  if  neither  were  used  (0). 

Compound  expressions  defined  once  (implementation  only). 

Repeated  compound  expressions  are  to  be  avoided  from  on  etfiosncy 
standpoint.  This  metric  is  based  on  the  number  of  compound  expressions 
which  appear  more  than  once. 

Number  of  overlays. 

The  use  of  overlays  requires  overhead  with  respect  to  processing  time. 
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Criteria: 


Effectiveness 


(.“5) 


(6) 


(7) 


Metric: 


(1) 


This  measure,  the  reciprocal  of  the  number  of  overlays,  reflects  that 
overhead.  It  can  be  applied  during  design,  when  the  overlay  scheme  is 
defined,  and  during  implementation. 

Free  of  bit/byte  packing/unpacking  in  loops. 

This  is  a  binary  measure  indicating  the  overhead  involved  in  bit/byte 
packing  and  unpacking.  P'acing  these  activities  within  loops  should  be 
avoided  if  possible. 

Module  linkages. 

This  measure  essentially  represents  the  inter-module  communication 
overhead.  The  measure  is  based  on  the  amount  of  execution  time  spent 
during  module-to-module  communication. 

Operating  system  linkages. 

This  measure  represents  the  module  to  OS  communication  overhead.  The 
measure  is  based  on  the  amount  of  execution  time  spent  during  module 
to  OS  communications. 

EF.3  Data  Usage  Efficiency  Measure. 

The  metric  at  the  module  level  is  the  sum  of  the  scores  of  the  following 
applicable  elements  divided  by  the  number  of  applicable  elements.  The 
system  metric  is  the  averaged  value  of  all  of  the  module  metric  values. 

Data  grouped  for  efficient  processing. 

The  data  utilized  by  any  module  is  to  be  organized  in  the  data  base, 
buffers,  arrays,  etc.,  in  a  manner  which  facilitates  efficient  processing. 
The  data  organization  during  design  and  implementation  is  to  be  exam¬ 
ined  to  provide  this  binary  measure. 
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Criteria: 


Effectiveness 


(2)  Variables  initialized  when  declared. 

This  measure  is  based  on  the  number  of  variables  used  in  a  module  which 
are  not  initialized  when  declared.  Efficiency  is  lost  when  variables  are 
initialized  during  execution  of  a  function  or  repeatedly  initialized  during 
iterative  processing. 

(3)  No  mix-mode  expressions. 

Processing  overhead  is  consumed  by  mix-mode  expressions  which  are 
otherwise  unnecessary.  This  measure  is  based  on  the  number  of  mix¬ 
mode  expressions  found  in  a  module. 

(4)  Common  choice  of  units/types. 

For  similar  reasons  as  expressed  in  (3)  above  this  convention  is  to  be 
followed.  The  measure  is  the  reciprocal  of  the  number  of  operations 
performed  which  have  uncommon  units  or  data  types. 

(5)  Data  indexed  or  referenced  for  efficient  processing. 

Not  only  the  data  organization,  (1)  above,  but  the  linkage  scheme 
between  data  items  effects  the  processing  efficiency.  This  is  a  binary 
measure  of  whether  the  indexing  utilized  for  the  data  was  chosen  to 
facilitate  processing. 

(6)  Static  data. 

This  metric  measures  the  numbers  of  data  items  which  were  referenced 
but  not  modified  during  execution. 

(7)  Dynamic  data. 

This  metric  measures  the  number  of  data  items  which  were  modified 
during  execution. 
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Criteria:  Effectiveness 

Metric:  EF.4  Storage  Efficiency  Measure. 

The  metric  at  the  moduie  level  is  the  sum  of  the  scores  of  the  following 
applicable  elements  divided  by  the  number  of  applicable  elements.  The 
metric  at  the  system  level  is  the  averaged  value  of  all  of  the  module 
metric  values. 

(1)  Storage  requirements  allocated  to  design. 

The  storage  requirements  for  the  system  are  to  be  allocated  to  the 
individual  modules  during  design.  This  measure  is  a  binary  measure  of 
whether  that  allocation  is  explicitly  made  (1)  or  not  (0). 

(2)  Virtual  storage  facilities  used. 

The  use  of  virtual  storage  or  paging  techniques  enhances  the  storage 
efficiency  of  a  system.  This  is  a  binary  measure  of  whether  these 
techniques  are  planned  for  and  used  (1)  or  not  (0). 

(3)  Common  data  defined  only  once. 

Often,  global  data  or  data  used  commonly  are  defined  more  than  once. 
This  consumes  storage.  This  measure  is  based  on  the  number  of  varia¬ 
bles  that  are  defined  in  a  module  that  have  been  defined  elsewhere. 


(4)  Program  segmentat'on. 

Efficient  segmentation  schemes  minimize  the  maximum  segment  length 
to  minimize  the  storage  requirement.  This  measure  is  based  on  the 
maximum  segment  length.  It  is  to  be  applied  during  design  when 
estimates  are  available  and  during  implementation. 

(5)  Dynamic  memory  management  utilized. 

This  is  a  binary  measure  emphasizing  the  advantages  of  using  dynamic 
memory  management  techniques  to  minimize  the  amount  of  storage 
required  during  execution.  This  is  planned  during  design  and  used  during 
implementation. 
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Criteria:  Effectiveness 


(6)  Data  packing  used. 

While  data  packing  was  discouraged  in  EF.2  (5)  in  loops  because  of  the 
overhead  it  adds  to  processing  time,  in  general  it  is  beneficial  from  a 
storage  efficiency  viewpoint.  This  binary  measure  applied  during  imple¬ 
mentation  recognizes  this  fact. 

(7)  Storage  optimizing  compiler/assembiy  language  used. 

This  binary  measure  is  similar  to  EF.2  (2)  except  from  the  viewpoint  of 
storage  optimization. 

(8)  Database  files/libraries  stored  at  only  one  node. 

Avoiding  multiple  files/libraries  increases  system  storage  optimization. 
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Criteria:  Functional  Overlap 


Metric:  FO.l  Functional  Overlap  Measure. 

This  metric  refers  to  the  overlap  of  functional  responsibility  or  computa¬ 
tion  between  the  two  systems  that  must  interoperate.  The  metric  is  the 
sum  of  the  scores  of  the  following  applicable  elements  divided  by  the 
number  of  applicable  elements. 

(1)  Number  of  duplicated  functions  in  the  system  that  are  to  interoperate. 

When  two  systems  must  be  made  to  interoperate,  functions  which  are 
duplicated  in  both  systems  must  be  examined  to  determine  any  potential 
conflict.  This  examination  for  function  conflict  will  require  additional 
effort  to  assess  the  two  functions  and  the  impact  each  may  have  on  the 
other  when  the  systems  interoperate. 

(2)  Number  of  duplicate  functions  to  be  deleted  in  one  or  the  other  system. 

The  presence  of  the  same  functions  being  implemented  or  accomplished 
in  both  systems  is  not  necessarily  detrimental  to  interoperability,  espec¬ 
ially  if  each  function  remains  independent  of  the  other  and  there  is  no 
need  to  communicate.  However,  if  one  of  the  systems  is  assigned  unique 
responsibility  for  that  function,  and  the  corresponding  function  is  to  be 
deleted  from  the  other  system,  then  the  amount  of  work  to  achieve 
iruer  operability  is  increased. 

(3)  Number  of  duplicated  function  pairs  to  be  synchronized. 

If  the  duplicated  functions  in  each  system  must  be  synchronized,  then 
the  effort  to  achieve  interoperation  wili  be  greater  than  that  in  (2) 
because  the  problems  of  synchronization  are  usually  more  complex  than 
those  of  deleting  one  function.  Various  timing,  format,  content,  and 
operational  considerations  may  arise  while  attempting  synchronization  of 
the  two  systems. 
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Criteria:  Functional  Overlap 


(4)  Number  of  duplicated  function  pairs  requiring  redundancy  management 
logic  to  combine  them. 

The  most  complex  resolution  of  duplicated  functions  is  the  use  of  a 
redundancy  management  scheme.  This  calls  not  only  for  intimate  com¬ 
munication  between  the  duplicated  functions,  but  also  calls  for  complex 
and  intricate  logic  to  resolve  apparent  differences,  identify  malfunctions, 
and  determine  and  implement  a  reconfiguration  approach. 


Criteria: 


Functional  Scope 


Metric: 


(1) 


(2) 


Metric: 


(1) 


(2) 


(3) 


FS.l  Function  Specificity 

The  degree  to  which  all  modules  in  the  system  perform  single  integral 
well  defined  functions.  The  metric  is  the  sum  of  the  scores  of  the 
following  applicable  elements  divided  by  the  number  of  applicable  ele¬ 
ments. 

Number  of  functions  performed  per  module. 

A  module  ideally  should  perform  a  single  integral  function.  This  mea¬ 
sure  io  based  on  the  number  of  functions  performed  in  a  module. 

Each  module  has  code  comments  about  functional  description. 

Comments  about  functions  performed  in  the  module  are  extremely  valu¬ 
able  to  the  person  who  wants  to  reuse  this  module.  The  measure  is 
based  on  the  number  of  modules  which  do  not  comply. 

FS.2  Function  Commonality 

This  metric  refers  to  the  usefulness,  to  other  applications,  of  the  func¬ 
tions  performed  by  the  software.  The  metric  is  the  sum  of  the  scores  of 
the  following  applicable  elements  divided  by  the  number  of  applicable 
e'ements. 

Is  the  function  constructed  in  a  manner  which  facilitates  or  encourages 
its  use  elsewhere  either  in  part  or  in  total? 

The  software  constructed  in  the  above  manner  tends  to  be  more  reus¬ 
able.  This  is  a  binary  measure. 

Are  the  input  quantities  well  defined? 

When  input  quantities  are  well  defined,  the  reuse  task  is  easier.  This  is 
a  binary  measure. 

Are  the  input  formats  well  defined? 

See  explanation  for  (2)  above. 
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Criteria:  Functional  Scope 


(4)  Are  the  outputs  or  database  well  defined  and  easy  to  interpret? 

A  similar  explanation  to  (2)  above  is  applicable  here. 

(5)  Does  the  function  performance  satisfy  one  of  the  specified  require¬ 
ments? 

This  is  an  application  dependent  metric. 

Metric:  FS.3  Function  Completeness 

The  degree  to  which  a  system  performs  a  total  function  in  terms  of  user 
need.  This  is  an  application  dependent  metric. 

(1)  Number  of  function  requirements  satisfied  in  the  specified  requirements. 
The  metric  is  the  number  of  user  requirements  satisfied  divided  by  the 
total  number  of  user  requirements.  The  value  is  computed  for  the 
system  metric. 
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Criteria:  Generality 

Metric:  GF..I  Module  Reference  By  Other  Modules. 

(1)  Number  of  modules  which  are  referenced  by  other  modules. 

This  metric  provides  a  measure  of  the  generality  of  the  modules  as  they 
are  used  in  the  current  system.  A  module  is  considered  to  be  more 
general  in  nature  if  it  is  used  (referenced)  by  more  thc.i  one  module. 
The  number  of  these  common  modules  divided  by  the  total  number  of 
modules  provides  the  measure. 

Metric:  GE.2  Implementation  for  Generality  Checklist. 

This  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 


(1)  Input,  processing,  output  functions  are  not  mixed  in  a  single  function. 

A  module  which  performs  I/O  as  well  as  processing  is  not  as  general  as  a 
module  which  simply  accomplishes  the  processing.  This  measure  is  based 
on  the  number  of  modules  that  violate  this  concept  at  design  and 
implementation. 


(2)  Applicat:  m  and  machine  dependent  functions  are  not  mixed  in  a  single 
module. 

Any  references  to  machine  dependent  functions  within  a  module  lessens 
its  generality.  An  example  would  be  referencing  the  system  clock  for 
timing  purposes.  This  measure  is  based  on  the  number  of  machine 
dependent  functions  in  a  module. 

(3)  Processing  not  data  volume  limited. 

A  module  which  has  been  designed  and  coded  to  accept  no  more  than 
100  data  item  inputs  for  processing  is  certainly  not  as  general  in  nature 
as  a  module  which  will  accept  any  volume  of  input.  This  measure  is 
based  on  the  number  of  modules  which  are  designed  or  implemented  to 
be  data  volume  limited. 
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Criteria:  Generality 


(4)  Processing  not  data  value  limited. 

A  previously  identified  element,  AM.2  (2)  of  Anomaly  Management  dealt 
with  checking  input  for  reasonableness.  This  capability  is  required  to 
prevent  providing  data  to  a  function  for  which  it  is  not  defined  or  its 
degree  of  precision  is  not  acceptable,  etc.  This  is  necessary  capability 
from  an  error  tolerance  viewpoint.  From  a  generality  viewpoint,  the 
smaller  the  subset  of  ail  possible  inputs  to  which  a  function  can  be 
applied  the  less  general  it  is.  Thus,  this  measure  is  based  on  the  number 
of  modules  which  are  data  value  limited. 
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Criteria: 

Metric: 

(1) 


(2) 


Metric: 

(1) 

(2) 


Independence 

ID.  I  Software  System  Independence  Measure 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

Dependence  on  software  system  utility  programs,  system  library  routines, 
and  other  system  facilities. 

The  more  utility  programs,  library  routines,  and  other  system  facilities 
that  are  used  within  a  system,  the  more  dependent  the  system  is  on  that 
software  system  environment.  A  SORT  utility  in  one  operating  system  is 
unlikely  to  be  exactly  similar  to  a  SORT  utility  in  another.  This 
measure  is  based  on  the  number  of  references  to  system  facilities  in  a 
module  divided  by  the  total  number  of  lines  of  code  in  the  module. 

Common,  standard  subset  of  language  used 

*’’’  use  of  nonstandard  constructs  of  a  language  that  may  be  available 
certain  compilers  cause  conversion  problems  when  the  software  is 
moved  to  a  new  software  system  environment.  This  measure  represents 
that  situation.  It  is  based  on  the  number  of  modules  which  are  coded  in 
a  'v  •'-standard  subset  of  the  language.  The  standard  subset  of  the 
jjage  is  to  be  established  during  design  and  adhered  to  during  imple¬ 
mentation. 

ID.2  Machine  Independence  Measure 

The  metric  is  the  sum  of  the  scores  of  thr.  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

Programming  language  used  available  on  other  machines. 

This  is  a  binary  measure  identifying  if  the  programming  language  used  is 
available  (1)  on  other  machines  or  not  (0).  This  means  the  same  version 
and  dialect  of  the  language. 

Free  from  input/output  references. 

Input  and  output  references  bind  a  module  to  the  current  machine 
configuration.  Thus  the  fewer  modules  within  a  system  that  contain 
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Criteria:  Independence 


input  and  output  references,  the  more  localized  the  problem  becomes 
when  conversion  is  considered.  This  measure  represents  that  fact  and  is 
based  on  the  number  of  I/O  references  within  a  module. 

(3)  Code  is  independent  of  word  and  character  size 

Instructions  or  operations  which  are  dependent  on  the  word  or  character 
size  of  the  machine  are  to  be  either  avoided,  or  parametric,  to  facilitate 
use  on  another  machine.  This  measure,  applied  to  the  source  code 
during  implementation,  is  based  on  the  number  of  modules  which  contain 
violations  to  the  concept  of  independence  of  word  and  character  size. 

(4)  Data  representation  machine  independent 

The  naming  conventions  (length)  used  are  to  be  standard  or  compatible 
with  other  machines.  This  measure  is  based  on  the  number  of  modules 
which  contain  variables  which  do  not  conform  to  standard  data  represen¬ 
tations. 
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Criteria: 


Modularity 


Metric 


:  MO. 2  Modular  Implementation  Measure. 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Hierarchical  structure. 

The  measure  refers  to  the  modular  implementation  of  the  top  down 
design  structure  mentioned  in  SI.l  (1).  The  hierarchical  structure  ob¬ 
tained  should  exemplify  the  following  rule:  interactions  between  mod¬ 
ules  are  retricted  to  flow  of  control  between  a  predecessor  module  and 
its  immediate  successor  modules.  This  measure  is  based  on  the  number 
of  violations  to  this  rule. 

(2)  Moduie  size  profile. 

The  standard  module  size  of  procedural  statements  can  vary.  100 
statements  has  been  mentioned  in  the  literature  frequently.  This  mea¬ 
sure  is  based  on  the  number  of  procedural  statements  in  a  module. 

(3)  Controlling  parameters  defined  by  calling  module. 

The  next  four  elements  further  elaborate  on  the  control  and  interaction 
between  modules  referred  to  by  (1)  above.  The  calling  module  defines 
the  controlling  parameters,  any  input  data  required,  and  the  output  data 
required.  Control  must  also  be  returned  to  the  calling  moduie.  This 
measure  is  based  on  the  number  of  calling  parameters  which  are  control 
parameters.  The  next  three  are  based  on  whether  a  rule  is  violated. 
They  can  all  be  measured  at  design  and  implementation. 

(4)  Input  data  controlled  by  calling  module. 

See  (3)  above. 

(5)  Output  data  provided  to  calling  module. 

See  (3)  above. 
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Criteria:  Modularit 


(6)  Control  returned  to  cailing  module. 

See  (3)  above. 

(7)  Modules  do  not  share  temporary  storage. 

This  is  a  binary  measure,  (1)  if  modules  do  not  share  temporary  storage 
and  (0)  if  they  do.  It  emphasizes  the  loss  of  module  independence  if 
temporary  storage  is  shared  between  modules. 

(S)  Each  module  represents  one  function. 

Ideally,  each  module  performs  only  one  function. 

Metric:  MO. 3  Modular  Design  Measure. 

The  metre  is  the  sum  of  the  scores  given  to  the  following  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Processes/functions/modules  have  loose  coupling. 

In  achieving  a  highly  modular  design  it  is  essential  to  minimize  the 
relationships  among  modules.  The  goal  is  to  design  modules  with  low 
coupling.  The  scale  of  coupling  from  worst  to  best  is:  1)  content 
coupling,  2)  common  coupling,  3)  external  coupling,  4)  control  coupling, 
5)  stamp  coupling,  and  6)  data  coupling. 

1)  Content  coupling  -  one  module  makes  reference  to  the  contents  of 
another  module. 

2)  Common  coupling  -  modules  reference  a  shared  global  data  struc¬ 
ture, 

3)  External  coupling  -  modules  reference  the  same  externally  declared 
symbol. 

4)  Control  coupling  -  one  module  passes  elements  of  control  as  argu¬ 
ments  to  another  module. 

5)  Stamp  coupling  -  two  modules  reference  the  same  data  structure, 
which  is  not  global. 

6)  Data  coupling  -  one  module  calls  another  and  the  modules  are  not 
coupled  as  defined  above  (in  1  through  5). 
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Criteria:  Modularity 


(2)  Processes/f unctions/ modules  have  high  cohesion. 

In  achieving  a  highly  modular  design  it  is  essential  to  maximize  the 
relationships  among  the  elements  of  each  module.  The  following  are 
relative  values  for  seven  types  of  cohesion: 


COHESION  TYPE 

VALUE 

7) 

Functional 

1.0 

6) 

Informational 

0.7 

5) 

Communicational 

0.5 

4) 

Procedural 

0.3 

3) 

Classical 

0.1 

2) 

Logical 

0.1 

1) 

Coincidental 

0.0 

The  fol  .owing  are  descriptions  of  the  seven  types  of  cohesion. 

1)  Coincidental 

.  No  meaningful  relationships  among  the  elements  of  a  module. 

.  Difficult  to  describe  the  module's  function(s). 

2)  Logical 

.  Module  performs  (at  each  invocation)  one  of  a  class  of  related 
functions  (e.g.,  "edit  ail  data”). 

.  Module  performs  more  than  one  function. 

3)  Classical 

.  Module  performs  one  of  a  class  of  functions  that  are  related  in 
time  (Program  procedure). 

.  Module  performs  more  than  one  function. 

4)  Procedural 

.  Module  performs  more  than  one  function,  where  the  functions 
are  related  with  respect  to  the  procedure  of  the  problem 
(Problem  procedure). 

5)  Communicational 

.  Module  has  procedural  strength;  in  addition,  all  of  the  elements 
"communicate”  with  one  other  (e.g.,  reference  same  data  or 
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Criteria:  Modularity 

pass  data  among  themselves). 

.  All  {unctions  use  the  same  data. 

6)  Informational 

.  Module  performs  multiple  functions  where  the  functions  (entry 
points  in  the  module)  deal  with  a  single  data  structure. 

.  Physical  packaging  together  of  two  or  more  modules  having  func¬ 
tional  strength. 

.  All  functions  use  the  same  data. 

7)  Functional 

.  All  module  elements  are  related  to  the  performance  of  a  single 
function. 


Reference: 

For  a  more  detailed  explanation  of  the  terms  used  to  describe  cohesion  and 
coupling  see  "Reliable  Software  Through  Composite  Design",  Myers, 
Glenford  3. 
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Criteria:  Operability 


Metric:  OP.  I  Operability  Checklist. 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  All  steps  of  operation  described  (normal  and  alternative  flows). 

This  binary  measure  identifies  whether  the  operating  characteristics  have 
been  described  in  the  requirements  specification,  and  if  this  description 
has  been  transferred  into  an  implementable  description  of  the  operation 
(usually  in  an  operator’s  manual).  The  description  of  the  operation 
should  cover  the  normal  sequential  steps  and  all  alternative  steps. 

(2)  All  error  conditions  and  responses  appropriately  described  to  operator. 

The  requirement  for  this  capability  must  appear  in  the  requirements 
specification,  must  be  considered  during  design,  and  coded  during  imple¬ 
mentation.  Error  conditions  must  be  clearly  identified  by  the  system. 
Legal  responses  for  all  conditions  are  to  be  either  documented  and/or 
prompted  by  the  system.  This  is  a  binary  measure  to  trace  the  evolution 
and  implementation  of  these  capabilities. 

(3)  Provisions  for  operator  to  interrupt,  obtain  operational  status,  save, 
modify,  and  continue  processing. 

The  capabilities  provided  to  the  operator  must  be  considered  during  the 
requirements  phase  and  then  designed  and  implemented.  Examples  of 
operator  capabilities  include  halt/resume  and  check  pointing.  This  is  a 
binary  measure  to  trace  the  evolution  of  these  capabilities. 

(4)  Number  of  operator  actions  reasonable  (requires  execution). 

The  number  of  operator  errors  can  be  related  directly  to  the  number  of 
actions  required  during  a  time  period.  This  measure  is  based  on  the 
amount  of  time  spent  requiring  manual  operator  actions  divided  by  the 
total  time  required  for  the  job. 

(5)  Job  set  up  and  tear  down  r  ocedures  described. 

The  specific  tasks  involved  in  setting  up  a  job  and  completing  it  are  to 
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be  described.  This  is  usually  documented  during  the  implementation 
phase  when  the  final  version  of  the  system  is  fixed.  This  is  a  binary 
measure  of  the  existence  of  that  description. 


(6)  Hard  copy  log  of  interactions  maintained. 

This  is  a  capability  that  must  be  planned  for  in  design  and  coded  during 
implementation.  It  assists  in  correcting  operational  errors,  improving 
efficiency  of  operation,  etc.  This  binary  measure  identifies  whether  it  is 
considered  in  the  design  and  imnlementation  phases  (1)  or  not  (0). 


(7)  Operator  messages  consistent  and  responses  standard. 

This  is  a  binary  measure  applied  during  design  and  implementation  to 
insure  that  the  interactions  between  the  operator  and  the  system  are 
simple  and  consistent.  Operator  responses  such  as  YES,  NO.  GO,  STOP, 
are  concise,  simple,  and  can  be  consistently  used  throughout  a  system. 
Lengthy,  differently  formatted  responses  nor  only  provide  difficulty  to 
the  operator  but  also  require  complex  error  checking  routines. 


(8)  Access  violations  and  responses  appropriately  described. 

Appropriate  decriptions  and  a  log  of  access  violations  will  enable  the 
operator  to  clearly  assess  the  system  status. 


(9)  Capability  for  operator  to  obtain  network  resource  status. 

This  capability  is  essential  for  managing  individual  nodes  resources  and 
for  providing  services  which  are  dependent  on  other  nodes. 


(1G)  Capability  to  select  different  nodes  for  different  types  of  processing  or 
for  different  types  of  information  retrieval. 

This  provision  expands  the  virtual  capability  and  versatility  of  the  node. 
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Criteria: 


Reconfigurability 


Metric:  RE. I  Restructure  Checklist. 

The  metric  is  the  sum  of  the  scores  given  to  the  following  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Configuration  of  communication  links  is  such  that  failure  of  one 
node/link  will  not  disable  communication  among  other  nodes. 

Alternate  communication  paths  ensure  the  ability  to  reconfigure  the 
network  in  the  event  of  a  single  point  failure. 

(2)  Specific  requirements  for  maintaining  data  base  integrity  under  anoma¬ 
lous  conditions. 

In  a  network  where  information  is  distributed  among  different  nodes,  and 
sometimes  duplicated  at  different  nodes,  it  is  essential  to  maintain  the 
integrity  of  the  total  database  when  conditions  are  non-normal. 

(3)  Provisions  for  maintaining  database  integrity  under  anomalous  conditions. 

A  scheme  is  required  for  implementing  the  requirements  referenced  in 
(2)  during  the  Preliminary  Design  phase. 

(4)  Node  can  rejoin  the  network  when  it  has  been  recovered. 

It  is  desirable  to  have  a  node  rejoin  the  network  without  interrupting 
basic  or  critical  network  functions. 

(5)  Data  replicated  at  two  or  more  distinct  nodes. 

Information,  especially  critical  data,  should  be  replicated  within  the 
■=ystem  to  insure  the  ability  to  reconfigure. 
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Criteria:  Self  Descriptiveness 


Metric:  SD.l  Quantity  of  Comments. 

The  metric  is  the  number  of  comment  lines  divided  by  the  total  number 
of  lines  in  each  module.  Blank  lines  are  not  counted.  The  average  value 
is  computed  for  the  system.level  metric. 

v 

(1)  Number  of  lines  of  source  code  and  non-blank  comments. 

Metric:  SD.2  Effectiveness  of  Comments  Measure. 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Modules  have  standard  formatted  prologue  comments. 

This  information  is  extremely  valuable  to  new  personnel  who  have  to 
work  with  the  software  after  development,  performing  maintenance, 
testing,  changes,  etc.  The  measure  at  the  system  level  is  based  on  the 
number  cf  modules  which  do  not  comply  with  a  standard  format  or  do 
not  provide  complete  information. 

(2)  Comments  set  off  from  code  in  uniform  manner. 

Blank  lines,  bordering  asterisks,  specific  card  columns  are  some  of  the 
techniques  utilized  to  aid  in  the  identification  of  comments.  The  meas¬ 
ure  is  based,  on  the  number  of  modules  which  do  not  follow  the  conven¬ 
tions  established  for  setting  off  the  comments. 

(3)  All  transfers  of  control  and  destinations  commented. 

This  form  of  comment  aids  in  the  understanding  and  ability  to  follow  the 
logic  of  the  module.  The  measure  is  based  on  the  number  of  modules 
which  do  not  comply. 

(4)  All  machine  dependent  code  commented. 

Comments  associated  with  machine  dependent  code  are  important  not 
only  to  explain  what  is  being  done  but  also  serves  to  identify  that 
portion  of  the  module  as  machine  dependent.  The  metric  is  based  on  the 
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number  of  modules  which  do  not  have  the  machine  dependent  code 
commented. 


(5)  All  non-standard  HOL  statements  commented. 

See  explanation  for  (4)  above. 

(6)  Attributes  of  all  declared  variables  commented. 

The  usage,  properties,  units,  etc.,  of  variables  are  to  be  explained  in 
comments.  The  measure  is  based  on  the  number  of  modules  which  do 
not  follow  this  practice. 

(7)  Comments  do  not  just  repeat  operation  described  in  language. 

Comments  are  to  describe  why,  not  what.  A  comment,  increment  A  by 
1,  for  the  statement  A=A-t-l  provides  no  new  information.  A  comment, 
increment  the  table  look-up  index,  is  more  valuable  for  understanding 
the  logic  of  the  module.  The  measure  is  based  on  the  number  of 
modules  in  which  comments  do  not  explain  the  why's. 

Metric:  SD.3  Descriptiveness  of  Language  Measure. 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  High  order  language  used. 

An  HOL  is  much  more  self-descriptive  than  assembly  language.  The 
measure  is  based  on  the  number  of  modules  which  are  implemented,  in 
whole  or  part,  in  assembly  or  machine  language. 

(2)  Variable  names  (mnemonics)  descriptive  of  physical  or  functional  pro¬ 
perty  represented. 

While  the  metric  appears  very  subjective,  it  is  quite  easy  to  identify  if 
variable  names  have  been  chosen  with  self-descriptiveness  in  mind. 


Criteria:  Self  Descriptiveness 


Three  variable  names  such  as  NAME,  POSIT,  SALRY  are  far  better  and 
more  easily  recognized  as  better  than  Al,  A2,  A3.  The  measure  is  based 
on  the  number  of  modules  which  do  not  utilize  descriptive  names. 

(3)  Source  code  logically  blocked  and  indented. 

Techniques  such  as  blocking,  paragraphing,  indenting  for  specific  con¬ 
structs  are  well  established  and  are  to  be  followed  uniformly  with  a 
system.  This  measure  is  based  on  the  number  of  modules  which  do  not 
comply  with  a  uniform  technique. 

(4)  One  statement  per  line. 

The  use  of  continuation  statements  and  multiple  statements  per  line 
causes  difficulty  in  reading  the  code.  The  measure  is  the  number  of 
continuations  plus  the  number  of  multiple  statement  lines  divided  by  the 
total  number  of  lines  for  each  module  and  then  averaged  over  all  of  the 
modules  in  the  system. 

(5)  Standard  format  for  organization  of  modules. 

All  modules  should  be  similar  in  structure  to  ease  understanding. 

(6)  No  language  keywords  used  dS  names. 

Names  should  be  unique  and  not  include  language  keywords. 
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Criteria:  Simp  'ic  it 


Metric:  SI.  1  Design  Structure  Measure. 

The  metric  is  the  sum  of  the  scores  of  the  applicable  elements  divided 
by  the  number  of  applicable  elements. 

(1)  Design  organized  in  top  down  fashion. 

A  hierarchy  chart  of  system  modules  is  usually  available  or  easy  to 
construct  from  design  documentation.  It  should  reflect  the  accepted 
notion  of  top  down  design.  The  system  is  organized  in  a  hierarchical 
tree  structure,  each  level  of  the  tree  represents  lower  levels  of  detail 
descriptions  of  the  processing. 

(2)  Module  independence. 

The  processing  done  within  a  module  is  not  to  be  dependent  on  the 
source  of  input  or  the  destination  of  the  output.  This  rule  can  be 
applied  to  the  module  description  during  design  and  the  coded  module 
during  implementation.  The  measure  for  this  element  is  based  on  the 
number  of  modules  which  do  not  comply  with  this  rule. 

(3)  Module  processing  not  dependent  on  prior  processing. 

The  proessing  done  within  a  module  is  not  to  be  dependent  upon  know¬ 
ledge  or  results  of  prior  processing,  e.g.,  the  first  time  through  the 
module,  the  nt’n  time  through,  etc.  This  rule  is  applied  as  above  at 
design  and  implementation. 

(4)  Each  module  description  includes  input,  output,  processing,  limitations. 
Documentation  which  describes  the  input,  output,  processing,  and  limita¬ 
tions  for  each  module  is  to  be  developed  during  design  and  available 
during  implementation.  The  measure  for  this  element  is  based  on  the 
number  of  modules  which  do  not  have  this  information  documented. 
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Criteria*. 


Simplicity 


(5)  Each  module  has  single  entrance,  single  exit. 

Determination  of  the  number  of  modules  that  violate  this  rule  at  design 
and  implementation  can  be  made  and  is  the  basis  for  the  metric. 

(6)  Size  of  data  base. 

The  size  of  the  data  base  in  terms  of  the  number  of  unique  data  items 
contained  in  the  data  base  relates  to  the  design  structure  of  the  soft¬ 
ware  system.  A  data  item  is  a  unique  data  element  for  example  an 
individual  data  entry  or  data  field. 

(7)  Compartmentalization  of  data  base 

The  structure  of  the  data  base  also  is  represented  by  its  modularization 
or  how  it  is  decomposed.  The  size  determined  in  (6)  above  divided  by 
the  number  of  data  sets  provided  this  measure.  A  data  set  corresponds 
to  the  first  level  of  decomposition  of  a  data  base,  e.g.,  a  set  in  a 
CODASYL  data  base,  a  record  in  a  file  system,  a  COMMON  in 
FORTRAN,  or  a  Data  Block  in  a  COMPOOL  system 

(8)  Programming  standard  developed. 

A  standard  for  programming  practices  will  enhance  uniformity  in  module 
development. 

(9)  Module  descriptions  include  identification  of  module  interfaces. 

Both  internal  and  external  interfaces  need  to  be  identified. 


Metric:  SI. 2  Structured  Language  or  Preprocessor. 

U)  Structured  language  or  preprocessor  used. 

The  use  of  a  structured  language  or  a  preprocessor  simplifies  the  pro¬ 
gramming  task. 
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Criteria: 


Simplicity 


Metric: 

(1) 

Metric: 

(1) 

(2) 

(3) 

(4) 


51. 3  Data  and  Control  Flow  Complexity  Measure 

Complexity  measure. 

(a)  Number  of  decision  points 

(b)  Number  of  branching  points 

The  metric  measure  is  the  reciprocal  of  the  number  branching  and 
decision  points. 

51. 4  Coding  Simplicity  Measure. 

The  metric  at  the  system  level  is  an  averaged  quantity  of  all  the  module 
measures  for  the  system.  The  module  measure  is  the  sum  of  the  scores 
of  the  following  applicable  elements  divided  by  the  number  of  applicable 
elements. 

Module  flow  top  to  bottom. 

This  is  a  binary  measure  of  the  logic  flow  of  a  module.  If  it  flows  top 
to  bottom,  it  is  given  a  value  of  1,  if  not  a  0. 

Negative  Boolean  or  complicated  compound  Boolean  expressions  used. 
Compound  expressions  involving  two  or  more  Boolean  operators  and  neg¬ 
ation  can  often  be  avoided.  These  types  of  expressions  add  to  the 
complexity  of  the  module.  The  measure  is  based  on  the  number  of  these 
complicated  expressions  per  executable  statement  in  the  module. 

Jumps  in  and  out  of  loops. 

Loops  within  a  module  should  have  one  entrance  and  one  exit.  This 
measure  is  based  on  the  number  of  loops  which  comply  with  this  rule 
divided  by  the  total  number  of  loops. 

Loop  index  modified. 

Modification  of  a  loop  index  not  only  complicates  the  logic  of  a  module 
but  causes  severe  problems  while  debugging.  This  measure  is  based  on 
the  number  of  loop  indices  which  are  modified  divided  by  the  tote  i 
number  of  loops. 
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Criteria: 


Simplicity 


(5)  Module  is  not  self-modifying. 

If  a  module  has  the  capability  to  modify  its  processing  logic  it  becomes 

very  difficult  to  recognize  what  state  it  is  in  when  an  error  occurs.  In 

addition,  static  analysis  of  the  logic  is  more  difficult.  This  measure 
emphasizes  the  added  complexity  of  self-modifying  modules. 

(6)  Number  of  statement  labels. 

This  measure  is  based  on  the  premise  .hat  as  more  statement  labels  are 
added  to  a  module  the  more  complex  it  becomes  to  understand. 

(7)  Nesting  level. 

The  greater  the  nesting  level  of  decisions  or  loops  within  a  module,  the 
greater  the  complexity.  The  measure  is  the  reciprocal  of  the  maximum 
nesting  level. 

(8)  Number  of  branches, 

The  more  paths  or  branches  that  are  present  in  a  module,  the  greater 

the  complexity.  This  measure  is  based  on  the  number  of  decision 

statements  per  executable  statements. 

(9)  Statement  simplicity  level. 

This  measure  is  based  on  the  number  of  declarative  and  data  manipula¬ 
tion  statements  per  executable  statement. 

(10)  Variable  mix  in  a  module. 

From  a  simplicity  viewpoint,  local  variables  are  far  better  than  global 
variables.  This  measure  is  the  ratio  of  internal  (local)  variables  to  total 
(internal  (local)  plus  external  (global))  variables  within  a  module. 

(11)  Variable  density. 

The  more  variables  used  in  a  module  the  greater  the  complexity  of  that 
module.  This  measure  is  based  on  the  number  of  variable  uses  in  a 
module  divided  by  the  maximum  possible  uses. 
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Criteria: 


Simplicity 


(.12)  Single  use  of  variables. 

Each  variable  should  have  a  singular  use. 

(13)  Code  written  according  to  programming  standard. 

Uniform  module  construction  and  coding  contentions  aid  in  minimizing 
complexity. 

(14)  Macros  and  subroutines  used  to  avoid  repeated  and  redundant  code. 

Use  of  macros  and  subroutines  is  yet  another  way  of  simplifying  code. 
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Criteria; 


Specificity 


Metric;  SP.i  Scope  of  Function  Measure. 

The  metric  is  the  sum  of  the  scores  given  to  the  following  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Input  density. 

The  fewer  the  input  parameters,  the  more  likely  the  module  is  singular 
in  function. 

(2)  Output  density. 

The  smaller  the  ratio  of  output  parameters  to  output  values,  the  more 
likely  the  module  is  singular  in  function. 

(3)  Same  function  cannot  be  accomplished  by  multiple  variant  forms. 

If  the  same  function  could  be  accomplished  by  multiple  different  mod¬ 
ules,  the  module  would  not  be  singular  in  function. 
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Criteria:  System  Accessibility 


Metric:  SA.  1  Access  Control  Checklist, 

The  metric  is  the  sum  ol  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  User  I/O  access  controls  provided. 

Requirements  for  user  access  control  must  be  identified  during  the 
requirements  phase.  Provisions  for  identification  and  password  checking 
must  be  designed  and  implemented  to  comply  with  the  requirements. 
This  binary  measure  identifies  whether  attention  has  been  placed  on  this 
area. 

(2)  Data  base  access  controls  provided. 

This  binary  measure  identifies  whether  requirements  for  data  base  con¬ 
trols  have  been  specified  and  designed  and  the  capabilities  implemented. 
Examples  of  data  base  access  controls  are  authorization  tables  and 
privacy  locks. 

(3)  Memory  protection  across  tasks  provided. 

Similar  to  (1)  and  (2)  above,  this  measure  identifies  the  progression  from 
a  requirements  statement  to  implementation  of  memory  protection 
across  tasks.  Examples  of  this  type  of  protection,  often  times  provided 
to  some  degree  by  the  operating  .system,  are  preventing  tasks  from 
invoking  other  tasks,  tasks  from  accessing  system  registers,  and  the  use 
of  privileged  commands. 

(4)  Network  access  controls  provided. 

Similar  to  the  above,  this  metric  identifies  the  need  for  access  control 
for  the  network  to  protect  both  the  operation  of  the  network  and 
individual  nodes. 
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Criteria:  System  Accessibility 

Metric:  SA.2  Access  Audit  Checklist. 

The  metric  is  the  sum  of  the  scores  given  to  the  following  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Provisions  for  recording  and  reporting  access  to  a  node. 

A  statement  of  the  requirement  for  this  type  capability  must  exist  in 
the  requirements  specification.  It  is  to  be  considered  in  the  design 
specification,  and  coded  during  implementation.  Examples  of  the  provi¬ 
sions  which  might  be  considered  would  be  the  recording  of  terminal  and 
processor  linkage,  data  file  accesses,  and  jobs  run  by  user  identification 
and  time* 

(2)  Provisions  for  immediate  indication  of  access  violations. 

In  addition  to  (1)  above,  access  audit  capabilities  required  might  include 
not  only  recording  accesses  but  immediate  identification  of  unauthorized 
accesses,  whether  intentional  or  not. 
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Criteria: 


System  Clarity 


Metric: 


(1) 


(2) 


(3) 


(4) 


(5) 


SC.l  Interface  Complexity. 

A  software  program  should  reduce  the  interface  complexity  and  promote 
the  system  clarity.  The  metric  is  the  sum  of  the  scores  of  the  following 
applicable  elements  divided  by  the  number  of  applicable  elements. 

Number  of  data  items  (variable  names)  used  to  specify  the  interface. 

The  measure  is  based  on  ine  number  of  data  items  specified  by  the 
interface. 

Number  of  data  items  passed  implicitly  across  interface  via  common 
global  data  without  adequate  comments. 

The  measure  is  based  on  the  number  of  data  items  which  are  passed 
implicitly  across  the  interface  and  without  adequate  comments  explana¬ 
tion. 

Number  of  nesting  levels  in  interface. 

The  greater  the  nesting  level  of  the  interface,  the  greater  the  interface 
complexity.  The  measure  is  the  reciprocal  of  the  number  of  nesting 
levels. 

Number  of  interlace  data  items  with  negative  qualification. 

The  procedures  returning  a  "TRUE"  upon  a  failure  tend  to  increase  the 
interface  complexity. 

Number  of  data  items  passed  across  moduie  interface  via  module  argu¬ 
ments  and  values  or  via  common  global  data. 

The  more  data  items  passed  across  the  interface  the  more  complex  the 
interface.  The  measure  is  the  reciprocal  of  the  number  of  data  items 
passed  across  the  interface. 
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(6)  Module  interfaces  established  by  common  control  blocks  or  common  data 
blocks  or  common  overlay  region  of  memory  or  common  I/O  devices  or 
global  variable  names  and  with  adequate  comments. 

The  interface  established  by  common  control  blocks  or  common  global 
data  is  more  complex  than  the  interface  established  by  parameter  lists. 
This  is  a  binary  measure. 

(7)  Modules  do  not  modify  other  modules. 

The  degree  of  coupling  is  higher  for  modules  that  modify  other  modules. 
The  measure  is  based  on  the  number  of  modules  which  do  not  comply 
with  the  rule. 

Metric:  SC. 2  Program  Flow  Complexity. 

Software  programs  should  reduce  the  program  flow  complexity  and  pro¬ 
mote  the  system  clarity.  The  metric  is  the  sum  of  the  scores  f  the 
following  applicable  elements  divided  by  the  number  of  applicable  ele¬ 
ments. 

(1)  Number  of  possible  unique  excution  paths. 

The  measure  is  the  reciprocal  of  the  number  of  unique  execution  paths. 

(2)  Number  of  IF  statements. 

The  measure  is  the  reciprocal  of  the  number  of  IF  statements. 

(3)  Number  of  function  CALLs  in  each  module. 

The  more  function  CALLs  are  present  in  a  module,  the  greater  the 
complexity.  The  measure  is  the  reciprocal  of  the  number  of  function 
CALLs. 

(4)  Number  of  control  variables  used  to  direct  execution  path  selection. 

The  measure  is  the  reciprocal  of  the  number  of  control  \  ariables. 
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Criteria: 


System  Clarity 


(5)  Number  of  DO  groups. 

The  measure  is  the  inverse  of  the  number  of  DO  groups. 


(6)  Each  module  has  code  comments  that  indicate  called-by  modules  and 
calling  modules. 

The  measure  is  based  on  the  number  of  modules  which  do  not  comply. 


Metric:  SC..3  Application  Functional  Complexity. 

Software  program  should  reduce  the  application  functional  complexity 
and  promote  the  system  clarity.  The  metric  is  the  sum  of  the  scores  of 
the  following  applicable  elements  divided  by  the  number  of  applicable 
elements. 


(1)  Separate  input/output  from  computational  functions. 

The  measure  is  based  on  the  number  of  modules  that  violate  this  rule. 

(2)  Modules  do  not  share  temporary  storage  locations. 

The  measure  is  based  on  the  number  of  modules  that  violate  this  rule. 

(3)  Separate  database-management  routines  and  storage-management  rou 
tines. 

The  measure  is  based  on  the  number  of  modules  that  violate  this  rule. 


(4)  Common  function  is  not  distributed  among  different  modules. 

Common  functions  distributed  among  several  different  modules  will  tend 
to  obscure  the  program  logic  in  each  module.  This  is  a  binary  measure. 

(5)  Module  is  not  made  to  do  too  many  (related  but  different)  functions. 

Too  many  related  but  different  functions  in  a  module  will  tend  to 
obscure  the  logic  with  tests  to  distinguish  among  the  various  functions. 
This  is  a  binary  measure. 
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Metric:  SC. 4  Communication  Complexity. 

Software  programs  should  reduce  the  communication  complexity  and 
promote  the  system  clarity.  The  metric  is  the  sum  of  the  following 
applicable  elements  divided  by  the  number  of  applicable  elements. 

(1)  Number  of  formal  parameters  each  routine. 

The  measure  is  the  number  of  parameters  divided  by  the  number  of 
global  variables. 

(2)  Common  global  variable  used  each  module. 

The  measure  is  the  reciprocal  of  the  number  of  common  global  variables 
used. 

(3)  Routine-Global-Routine  data  binding. 

The  measure  is  based  on  the  number  of  global  variables  which  are 
modified  by  one  routine  and  referenced  to  other  routines. 

(4)  Module  connections  are  established  by  referring  to  other  modules  by 
their  functional  names,  not  internal  elements  of  other  modules. 

Modules  whose  connections  are  established  by  referring  to  other  modules 
by  their  functional  names  are  more  loosely  coupled  than  are  modules 
whose  connections  refer  to  internal  elements  of  other  modules.  The 
measure  is  based  on  the  number  of  modules  which  do  not  comply. 

(5)  Ccmmunication  between  modules  is  by  passing  data,  not  by  passing 
control  elements. 

The  measure  is  based  on  the  number  of  modules  which  do  not  comply. 
Metric:  SC. 5  Structure  Clarity 

To  remove  the  program  impurities,  to  improve  the  structure  clarity,  and 
make  software  easier  to  understand.  The  metric  is  a  measure  reflecting 
this  improvement  and  is  the  sum  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 


C-72 


Criteria:  System  Clarit 


(1)  Do  not  compute  the  same  value  more  than  once. 

Whenever  a  specific  combination  of  terms  must  be  used  more  than  once 
a  new  name  should  be  assigned  to  that  combination  and  that  new  name 
should  be  utilized  in  the  subsequent  occurrences  of  that  term.  The 
binary  metric  measure  reflects  this  readability  improvement. 

(2)  Do  not  insert  a  statement  which  never  needs  to  be  executed. 

To  remove  the  unwarranted  assignment  statement  and  improve  the  com¬ 
prehensibility  of  program.  This  is  a  binary  measure  to  reflect  this 
improvement. 

(3)  Maintain  a  constant  meaning  for  each  variable. 

Modules  should  not  use  the  same  variable  to  represent  different  types  of 
values  in  different  portions  of  program  to  improve  the  understandability. 
Tins  is  a  binary  measure  to  reflect  this  improvement. 

W  Eliminate  unnecessary  intermediate  variables. 

See  explanation  ?or  (2)  above. 
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Criteria: 


System  Compatibility 


Metric: 


(1) 


(2) 


(3) 


(4) 


Metric: 


(1) 


SY.l  Communication  Compatibility  Checklist. 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

Same  I/O  transmission  rates  in  both  systems. 

If  the  two  systems  have  incompatible  transmission  rates,  extra  effort 
will  be  required  to  avoid  buffer  overruns,  data  overruns,  and  lost  data. 
Thus,  the  effort  to  interoperate  in  this  case  is  increased. 

Same  communication  protocol  in  both  systems. 

Compatible  communication  protocols  assures  the  systems  can  begin  to 
converse.  If  the  protocols  are  incompatible,  then  additional  work  will  be 
required  so  that  the  systems  can  initiate  mutual  communication. 

Same  message  content  in  both  systems. 

If  the  content  of  the  messages  are  not  the  same,  that  is,  the  same  units, 
the  same  variable,  the  same  reference  points,  and  the  same  reference 
structure,  then  the  message  will  have  a  meaning  to  the  receiver  differ¬ 
ent  from  that  intended  by  the  sender. 

Same  message  structure  and  sequence  in  both  systems. 

Even  though  the  protocols  may  be  compatible,  and  the  data  of  mutual 
format  and  type,  interoperation  may  be  impossible  if  the  message  struc¬ 
ture  and  message  sequences  are  not  compatible. 

SY.2  Data  Compatibility  Checklist. 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

Is  data  in  both  systems  in  the  same  format  (ASCII,  EBCDIC,...) 

The  format  of  the  data  transmitted  between  the  systems  should  be 
identical,  otherwise,  additional  effort  must  be  spent  converting  the 
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format  in  one  system;  or  a  hardware  or  software  reformatter  must  be 
designed  and  implemented. 


(2)  Same  data  base  structure  in  both  systems. 

If  the  data  base  structures  are  compatible,  then  consistent  accessing  and 
indexing  interpretations  are  possible,  lessening  the  chance  of  incom¬ 
patibilities  which  would  increase  the  effort  to  achieve  interoperation. 


(3)  Same  data  base  access  techniques  in  both  systems. 

This  metric  component  is  related  to  (2),  but  it  is  unique  in  that  it 
assures  that  the  accessing  variables  will  be  as  similar  as  possible 
between  the  systems,  reducing  the  conversion  necessary  between  sys¬ 
tems. 

Metric:  SY.3  Hardware  Compatibility  Checklist. 

The  metric  is  xhe  sum  of  the  scores  given  to  the  following  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Same  word  length  in  both  systems. 

If  both  systems  use  the  same  standard  word  length,  then  problems  of 
differing  accuracy  and  conversion  are  removed. 

(2)  Same  interrupt  structure  in  both  systems. 

If  both  systems  use  computers  with  the  same  interrupt  structure,  it  is 
likely  that  they  will  be  mutually  compatible  in  their  interfaces  with  the 
real  world  of  sensors,  etc. 

(3)  Same  instruction  set  in  both  systems. 

If  both  systems  use  computers  with  identical  instruction  sets,  then  they 
truly  "talk  the  same  language."  This  compatibility  should  contribute  to 
reduced  effort  to  achieve  interoperation  between  the  two  systems. 
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System  Compatibility 


Metric: 

(1) 

(2) 

(3) 


Metric: 

(1) 


SY.4  Software  Compatibility  Checklist. 

The  metric  is  the  sum  of  the  scores  given  to  the  following  elements 
divided  by  the  number  of  applicable  elements. 

Same  source  language  in  both  systems. 

If  the  source  language  used  in  the  two  systems  is  the  same,  then  many 
compatibilities  are  already  provided;  if  not,  the  effort  to  interoperate 
will  increase  due  to  resolution  of  language  feature  discrepancies. 

Same  operating  system  in  both  systems. 

Identical  operating  systems  will  provide  assurance  of  consistent  features 
and  methods  of  operation.  Thus,  the  effort  required  to  interoperate 
should  be  reduced. 

Same  support  software  in  both  systems. 

If  identical  support  software  is  used  for  the  systems  that  must  inter¬ 
operate,  it  is  likely  that  both  may  be  constructed  in  much  the  same  way. 
The  communication  necessary  to  service  both  systems  will  be  simplified. 
Finally,  duplicate  support  software  centers  may  provide  greater  reli¬ 
ability,  or,  alternatively,  the  possibility  for  cost  reductions. 

SY.5  Documentation  for  Other  Systems. 

Is  the  other  system  documentation  available  in  a  form  that  is  up-to-date, 
complete,  and  clearly  organized  and  written? 

Many  questions  about  the  other  system  will  arise  in  achieving  interoper¬ 
ability,  and  the  most  efficient  and  practical  way  of  answering  them  is 
the  availability  of  documentation  on  the  other  system.  For  the  docu¬ 
mentation  to  be  useful,  however,  it  must  meet  certain  requirements.  It 
must  reflect  the  other  system  as  it  currently  exists,  or  as  it  will  exist  at 
the  time  of  interoperation;  so  the  documentation  must  be  up-to-date. 
The  documentation  must  also  be  complete,  rt  least  to  the  extent  neces¬ 
sary  to  answer  ail  questions  relating  to  interoperabiiity.  But,  even 
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the  most  complete  and  up-to-date  documents  will  be  relatively  useless  if 
they  are  not  clearly  organized  and  clearly  written.  The  reader  must  be 
able  to  find  his  way  efficiently  to  the  answer  he  needs,  and  when  found, 
the  answer  must  be  stated  clearly.  Otherwise,  the  time  lost  to  locate 
and  understand  the  information  will  be  excessive  and  it  is  likely  the 
reader  will  make  an  assumption  for  his  purposes.  Once  again,  the  result 
is  likely  to  be  additional  interoperability  problems. 


Criteria: 


Traceability 


Metric:  TF.l  Cross  Reference. 

(1)  Cross  reference  relating  functions/modules  to  requirements. 

During  design,  the  icentification  of  which  itemized  requirements  are 
satisfied  in  the  design  of  a  module  are  documented.  A  traceability 
matrix  is  an  example  of  how  this  can  be  done.  During  implementation, 
which  itemized  requirements  are  being  satisfied  by  the  module  imple¬ 
mentation  are  to  be  identified.  Some  form  of  automated  notation, 
prologue  comments  or  imbedded  comments,  is  used  to  provide  this  cross 
reference.  The  binary  metric  is  the  identification  of  a  tracing  of 
requirements  into  design  and  into  code. 


Criteria: 


Training 


Metric:  TN.l  Training  Checklist. 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Lesson  pians/training  material  developed  for  operators,  end  users,  main- 
tainers. 

This  is  a  binary  measure  of  whether  this  type  documentation  is  provided 
during  the  implementation  phase. 

(2)  Realistic  simulated  exercises  provided. 

This  is  a  binary  measure  of  whether  exercises,  which  represent  the 
operational  environment,  are  developed  during  the  implementation  phase 
for  use  in  training. 

(3)  Sufficient  ’help1  and  diagnostic  information  available  on-line. 

This  is  a  binary  measure  of  whether  the  capability  to  aid  the  operator  in 
familiarization  with  the  system  has  been  designed  and  built  into  the 
system.  Provision  of  a  list  of  legal  commands  or  a  list  of  the  sequential 
steps  involved  in  a  process  are  examples. 

W  Selectable  levels  of  aid  and  guidance  for  users  of  different  degrees  of 
expertise. 

This  is  a  binary  measure  of  multi-level  capability  for  user  familiariza¬ 
tion. 
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Virtuality 


Metric 


:  VR.l  System/Data  Independence  Checklist. 

This  metric  is  the  sum  of  the  scores  given  to  the  following  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Specific  requirements  for  virtual  storage  structure. 

Requiring  a  virtual  storage  structure  is  the  key  to  providing  the  user 
with  a  virtual  system. 

(2)  Provisions  for  virtual  storage  structure  (user  can  obtain  data  without 
knowing  identity/location  of  storage  device). 

During  Preliminary  Design,  a  scheme  is  required  to  implement  the 
requirements  referenced  in  (1).  The  scheme  may  be  elaborate  if  data  is 
widely  distributed  within  the  network. 

(3)  Users  can  manipulate  data  as  if  it  were  not  replicated  elsewhere  in  the 
system. 

This  measure  refers  to  potential  configuration  management  problems  in  a 
network  where  the  same  data  is  replicated  at  different  nodes. 

(4)  Each  user  can  utilize  the  system  as  though  it  were  dedicated  to  that 
user. 

Presenting  each  user  with  a  system  which  is  virtually  dedicated  to  that 
user  maximizes  the  capabilities  available  to  the  user. 

(5)  User  is  presented  with  a  complete  logical  system  without  regard  to 
physical  topology. 

Lifting  the  requirement  for  the  user  to  know  the  physical  topology  of 
the  system  simplifies  the  user's  task  with  respect  to  the  system. 
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Criteria: 


Visibility 


Metrics  VS.l  Module  Testing  Measure. 

The  system  level  metric  is  an  average  of  all  module  measures.  The 
module  measure  is  the  sum  of  the  scores  given  to  the  following  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Path  coverage. 

Plans  for  testing  the  various  paths  within  a  module  should  be  made 
during  design  and  test  cases  actually  developed  during  implementation. 
This  measure  identifies  the  number  of  paths  planned  to  be  tested  divided 
by  the  total  number  of  paths. 

(2)  Input  parameters  boundary  tested. 

The  other  aspect  of  module  testing  involves  testing  the  input  ranges  to 
the  module.  This  is  done  by  exercising  the  module  at  the  various 
boundary  values  of  the  input  parameters.  Plans  to  do  this  must  be 
specified  during  design  and  coded  during  implementation.  The  measure 
is  the  number  of  parameters  to  be  boundary  tested  divided  by  the  total 
number  of  parameters. 


Metric:  VS. 2  Integration  Testing  Measure. 

The  metric  is  the  sum  of  the  scores  given  to  the  following  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Module  interfaces  tested. 

One  aspect  of  integration  testing  is  the  testing  of  all  module-to-moduie 
interfaces.  Plans  to  accomplish  this  testing  are  prepared  during  design 
and  the  tests  are  developed  during  implementation.  The  measure  is 
based  on  the  number  of  interfaces  to  be  tested  divided  by  the  total 
number  of  interfaces. 
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(2)  Performance  requirements  (timing  and  storage)  coverage. 

The  second  aspect  of  integration  testing  involves  checking  for  com¬ 
pliance  at  the  module  and  subsystem  level  with  the  performance  require¬ 
ments.  This  testing  is  planned  during  design  and  the  tests  are  developed 
during  implementation.  The  measure  is  the  number  of  performance 
requirements  to  be  tested  divided  by  the  total  number  of  performance 
requirements. 

Metric:  VS. 3  System  Testing  Measure. 

The  metric  is  the  sum  of  the  scores  given  to  the  following  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Module  coverage  (for  all  test  scenarios). 

One  aspect  of  system  vesting  which  can  be  measured  as  early  as  the 
design  phase  is  the  equivalent  to  path  coverage  at  the  module  level.  For 
all  system  test  scenarios  planned,  the  percent  of  ail  of  the  modules  to 
be  exercised  is  important. 

(2)  Identification  of  test  inputs  and  outputs  in  summary  form. 

The  results  of  tests  and  the  manner  in  which  these  results  are  displayed 
are  very  important  to  the  effectiveness  of  testing.  This  is  especially 
true  during  system  testing  because  of  the  potentially  large  volume  of 
input  and  output  daU.  This  binary  measure  simply  identifies  if  the 
capability  exists  to  display  test  inputs  and  outputs  in  a  summary  fashion. 
The  measure  can  be  applied  to  the  plans  and  specifications  in  the  design 
phase  and  the  development  of  this  capability  during  implementation. 


MISSION 

of 

Rome  Air  Development  Center 

RAVC  plans  and  executes  research,  de.ve.Lopme.nt,  test  and 
selected  acquisition  programs  in  support  of,  Command,  Control 
Communications  and  In.teltigc.nce  (C5I )  activities.  Technical 
and  engineering  support  within  areas  of  technical  competence 
is  provided  to  ESP  Program  Offices  IPOs)  and  other  ESV 
elements.  The  principal  technical  mission  areas  axe 
communications,  electromagnetic  guidance  and  control,  sur¬ 
veillance  of  ground  and  aerospace  objects ,  intelligence  data, 
collection  and  handling,  information  system  technology, 
ionospheric  paopagati.cn,  solid  state  sciences,  mtctotmvp. 
physics  and  electronic  reliability,  maintainability  and 
compatibil.ity . 


